Nov 02 13:40:43.756877 ip-10-0-139-26 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 13:40:43.756888 ip-10-0-139-26 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 13:40:43.756895 ip-10-0-139-26 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 13:40:43.757117 ip-10-0-139-26 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 13:40:54.006651 ip-10-0-139-26 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 13:40:54.006667 ip-10-0-139-26 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 56f3a6b22d884534bce7ba50c0584984 --
Nov 02 13:42:47.106088 ip-10-0-139-26 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 13:42:47.471892 ip-10-0-139-26 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:42:47.471892 ip-10-0-139-26 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 13:42:47.471892 ip-10-0-139-26 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:42:47.471892 ip-10-0-139-26 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:42:47.477151 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477065    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:47.477151 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477151    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477155    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477159    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477163    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477166    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477168    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477171    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477174    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477177    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477179    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477182    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477184    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477187    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477190    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477192    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477195    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477197    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477200    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477206    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477209    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:47.477215 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477212    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477215    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477217    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477220    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477223    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477226    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477229    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477232    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477234    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477237    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477239    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477242    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477245    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477248    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477252    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477257    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477259    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477262    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477265    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:47.477679 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477267    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477270    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477272    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477275    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477277    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477280    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477283    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477285    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477287    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477290    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477292    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477297    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477301    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477305    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477307    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477310    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477313    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477316    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477318    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477321    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:47.478139 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477323    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477327    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477329    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477332    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477334    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477337    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477340    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477342    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477345    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477348    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477350    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477353    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477356    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477358    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477361    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477363    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477366    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477369    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477371    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477374    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:47.478617 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477376    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477379    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477382    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477384    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477388    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477391    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477393    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477396    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477398    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477401    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477403    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477406    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477409    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477412    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477414    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477417    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477419    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477423    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477426    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:47.479096 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477428    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477431    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477433    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477436    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477438    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477441    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.477444    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478281    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478290    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478294    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478297    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478300    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478303    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478306    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478309    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478311    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478314    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478317    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478320    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:47.479539 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478323    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478327    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478329    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478332    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478335    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478337    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478340    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478343    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478345    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478348    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478352    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478356    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478360    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478363    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478366    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478368    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478371    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478373    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478376    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:47.480053 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478379    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478381    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478384    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478387    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478389    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478392    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478394    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478397    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478405    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478415    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478418    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478421    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478424    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478433    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478436    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478439    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478442    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478444    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478447    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478450    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:47.480509 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478452    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478454    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478457    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478459    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478462    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478464    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478467    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478469    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478472    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478475    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478478    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478481    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478483    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478486    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478488    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478491    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478494    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478496    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478499    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478501    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:47.481008 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478504    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478506    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478509    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478511    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478514    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478516    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478524    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478527    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478530    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478532    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478535    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478538    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478540    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478542    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478545    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478547    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478550    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478552    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478555    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478557    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:47.481486 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478560    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478562    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478564    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478567    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478570    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478572    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478575    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478578    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478580    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478582    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478585    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478589    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478593    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478596    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.478599    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479091    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479097    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479100    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479104    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:47.481975 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479107    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479109    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479112    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479115    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479118    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479120    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479123    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479126    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479128    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479131    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479133    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479136    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479139    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479142    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479144    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479148    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479151    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479154    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479156    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:47.482438 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479159    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479161    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479164    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479166    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479169    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479171    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479174    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479176    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479179    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479181    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479184    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479187    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479190    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479193    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479195    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479198    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479201    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479203    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479206    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479208    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:47.482922 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479211    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479213    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479216    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479218    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479221    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479224    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479227    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479229    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479232    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479235    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479237    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479240    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479242    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479245    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479247    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479250    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479252    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479255    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479257    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479260    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:47.483397 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479262    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479265    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479267    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479270    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479272    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479275    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479278    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479281    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479283    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479286    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479288    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479291    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479294    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479296    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479298    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479301    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479303    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479308    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479312    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:47.483884 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479315    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479318    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479322    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479325    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479327    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479330    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479333    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479335    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479338    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479341    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479343    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479346    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479348    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479352    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479354    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479357    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479359    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479362    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479364    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479367    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:47.484337 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479370    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479372    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479375    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.479377    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480455    2582 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480470    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480477    2582 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480482    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480486    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480490    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480495    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480499    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480502    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480505    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480511    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480515    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480518    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480521    2582 flags.go:64] FLAG: --cgroup-root=""
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480524    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480528    2582 flags.go:64] FLAG: --client-ca-file=""
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480530    2582 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480534    2582 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480537    2582 flags.go:64] FLAG: --cluster-domain=""
Nov 02 13:42:47.484857 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480540    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480543    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480546    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480550    2582 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480553    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480556    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480560    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480563    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480566    2582 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480569    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480572    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480576    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480579    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480583    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480587    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480590    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480592    2582 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480595    2582 flags.go:64] FLAG: --enable-server="true"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480599    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480603    2582 flags.go:64] FLAG: --event-burst="100"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480607    2582 flags.go:64] FLAG: --event-qps="50"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480609    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480612    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480617    2582 flags.go:64] FLAG: --eviction-hard=""
Nov 02 13:42:47.485406 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480621    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480624    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480627    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480630    2582 flags.go:64] FLAG: --eviction-soft=""
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480633    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480636    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480638    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480641    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480644    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480647    2582 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480650    2582 flags.go:64] FLAG: --feature-gates=""
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480654    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480657    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480660    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480663    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480666    2582 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480669    2582 flags.go:64] FLAG: --help="false"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480672    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480675    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480678    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480682    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480685    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480689    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480692    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 13:42:47.486048 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480695    2582 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480698    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480701    2582 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480703    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480706    2582 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480710    2582 flags.go:64] FLAG: --kube-reserved=""
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480713    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480717    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480720    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480723    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480726    2582 flags.go:64] FLAG: --lock-file=""
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480729    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480732    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480735    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480744    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480747    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480751    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480753    2582 flags.go:64] FLAG: --logging-format="text"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480756    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480760    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480763    2582 flags.go:64] FLAG: --manifest-url=""
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480765    2582 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480770    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480773    2582 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480787    2582 flags.go:64] FLAG: --max-pods="110"
Nov 02 13:42:47.486612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480790    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480793    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480796    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480800    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480803    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480806    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480809    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480816    2582 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480819    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480822    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480825    2582 flags.go:64] FLAG: --pod-cidr=""
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480828    2582 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480831    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480833    2582 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480838    2582 flags.go:64] FLAG: --port="10250"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480841    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480844    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09f42ae1c1b5205ce"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480847    2582 flags.go:64] FLAG: --qos-reserved=""
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480850    2582 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480853    2582 flags.go:64] FLAG: --register-node="true"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480856    2582 flags.go:64] FLAG: --register-with-taints=""
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480860    2582 flags.go:64] FLAG: --registry-burst="10"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480863    2582 flags.go:64] FLAG: --registry-qps="5"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480866    2582 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480868    2582 flags.go:64] FLAG: --reserved-memory=""
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480872    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 13:42:47.487234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480875    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480878    2582 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480881    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480884    2582 flags.go:64] FLAG: --runonce="false"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480887    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480890    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480892    2582 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480895    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480898    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480901    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480905    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480908    2582 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480911    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480914    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480917    2582 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480919    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480922    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480925    2582 flags.go:64] FLAG: --system-cgroups=""
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480928    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480934    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480938    2582 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480941    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480945    2582 flags.go:64] FLAG: --tls-min-version=""
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480948    2582 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480951    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 13:42:47.487927 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480953    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480956    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480959    2582 flags.go:64] FLAG: --v="2"
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480964    2582 flags.go:64] FLAG: --version="false"
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480968    2582 flags.go:64] FLAG: --vmodule=""
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480972    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.480975    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481103    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481107    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481110    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481112    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481115    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481117    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481122    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481125    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481129    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481132    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481135    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481138    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481141    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:47.488525 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481144    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481146    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481149    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481151    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481154    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481157    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481160    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481164    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481166    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481169    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481172    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481175    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481177    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481180    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481182    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481185    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481188    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481191    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481193    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481196    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:47.489306 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481198    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481201    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481204    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481206    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481209    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481212    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481214    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481219    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481221    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481224    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481226    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481229    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481232    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481234    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481237    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481240    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481244    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481247    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481251    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:47.490065 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481255    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481258    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481260    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481263    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481266    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481268    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481271    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481274    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481276    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481279    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481282    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481284    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481287    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481290    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481292    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481295    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481297    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481300    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481302    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481305    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:47.490530 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481309    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481311    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481314    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481317    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481319    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481322    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481325    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481327    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481330    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481332    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481335    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481340    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481345    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481348    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481350    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481353    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481355    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481358    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481360    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481363    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:47.491027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481365    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481368    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481370    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481373    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481375    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481378    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481380    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481383    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481386    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481388    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481391    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481393    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481397    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481399    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.481425    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481458    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:47.491510 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481462    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481465    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481469    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481472    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481475    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481477    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481480    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481483    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481486    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481489    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481492    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481494    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481498    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481500    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481503    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481505    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481508    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481511    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481513    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:47.491983 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481516    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481518    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481521    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481523    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481526    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481529    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481533    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481535    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481538    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481540    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481544    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481546    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481549    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481552    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481554    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481557    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481560    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481563    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481565    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:47.492425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481568    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481570    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481573    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481576    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481578    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481581    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481583    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481586    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481589    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481591    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481594    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481596    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481599    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481601    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481604    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481606    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481609    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481611    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481614    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481616    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:47.492883 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481619    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481621    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481624    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481627    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481630    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481632    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481635    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481637    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481640    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481644    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481647    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481651    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481654    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481657    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481660    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481662    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481665    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481667    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481670    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:47.493345 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481672    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481675    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481677    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481680    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481682    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481685    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481687    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481690    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481692    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481695    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481697    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481700    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481702    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481705    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481707    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481710    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481714    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481716    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481719    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481722    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:47.493864 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481725    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481727    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481730    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481732    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481736    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481738    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481741    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.481744    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.490337    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.490352    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.490983    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.490990    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491053    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491059    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491063    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491066    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491069    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491072    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491075    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491077    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:47.494370 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491080    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491083    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491086    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491089    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491091    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491094    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491097    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491099    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491102    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491105    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491108    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491111    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491113    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491116    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491119    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491127    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491130    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491133    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491136    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491139    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:47.494887 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491141    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491144    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491147    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491150    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491153    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491156    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491158    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491161    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491164    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491166    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491169    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491171    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491174    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491176    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491179    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491182    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491184    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491187    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491190    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:47.495353 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491193    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491195    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491198    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491200    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491203    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491206    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491209    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491212    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491215    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491217    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491220    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491223    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491226    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491228    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491231    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491234    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491236    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491239    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491242    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:47.495795 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491244    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491247    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491250    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491252    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491255    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491258    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491260    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491263    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491265    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491268    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491271    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491273    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491276    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491279    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491282    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491285    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491287    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491290    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491292    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:47.496250 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491295    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491297    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491300    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491302    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491305    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491308    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491311    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491314    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491316    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491318    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491321    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491324    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491327    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491331    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491334    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491337    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491340    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491343    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491347    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:47.496690 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491351    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491354    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.491382    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491518    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491523    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491526    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491529    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491533    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491536    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491539    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491541    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491544    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491547    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491549    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491552    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:47.497171 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491555    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491559    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491563    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491566    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491568    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491571    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491573    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491576    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491579    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491582    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491584    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491587    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491589    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491592    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491594    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491597    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491599    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491602    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491604    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:47.497555 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491607    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491610    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491612    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491615    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491617    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491620    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491622    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491626    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491628    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491631    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491634    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491636    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491639    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491641    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491644    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491647    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491649    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491652    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491654    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491657    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:47.498027 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491659    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491662    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491664    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491667    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491670    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491673    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491676    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491678    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491681    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491684    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491687    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491689    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491692    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491694    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491697    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491700    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491702    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491705    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491707    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491710    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:47.498548 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491713    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491716    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491718    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491721    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491724    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491726    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491729    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491731    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491734    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491736    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491739    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491742    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491744    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491748    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491751    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491754    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491757    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491760    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491763    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491766    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:47.499101 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491770    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491773    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491788    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491793    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491796    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491799    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491802    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491804    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491807    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491809    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491812    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491815    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491817    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491821    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491823    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.491848    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:42:47.499562 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491880    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491885    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491888    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491891    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491893    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491896    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491898    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491901    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491903    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491906    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491908    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491911    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491913    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491916    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491919    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491921    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491925    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491928    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491931    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491934    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:42:47.499986 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491936    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491939    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491941    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491944    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491946    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491949    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491953    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491956    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491960    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491963    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491966    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491968    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491971    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491974    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491976    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491979    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491981    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491984    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491986    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:42:47.500455 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491989    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491992    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491994    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.491997    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492000    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492002    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492005    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492009    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492012    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492016    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492019    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492021    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492024    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492026    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492029    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492032    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492034    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492037    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492039    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:42:47.500931 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492042    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492044    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492047    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492056    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492060    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492063    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492066    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492068    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492071    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492074    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492077    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492079    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492082    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492084    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492087    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492089    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492092    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492095    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492097    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:42:47.501376 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492100    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492105    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492108    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492112    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492114    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492117    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492119    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492122    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492125    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492127    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492130    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492132    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492135    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492138    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492141    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492143    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492146    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492148    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492151    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492156    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:42:47.501899 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492158    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492161    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492164    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492167    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492169    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492172    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492175    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492177    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:47.492180    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.492234    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.496654    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.497649    2582 server.go:1014] "Starting client certificate rotation"
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.497741    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.497791    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.499940    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:42:47.502423 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.501822    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:42:47.512108 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.512095    2582 log.go:25] "Validated CRI v1 runtime API"
Nov 02 13:42:47.515887 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.515871    2582 log.go:25] "Validated CRI v1 image API"
Nov 02 13:42:47.516680 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.516655    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 13:42:47.520570 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.520546    2582 fs.go:135] Filesystem UUIDs: map[5e50ca4f-00de-4ba8-ba44-cfa240489609:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9e0b04ea-c5a0-410d-809f-df9fefd195af:/dev/nvme0n1p4]
Nov 02 13:42:47.520570 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.520562    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 13:42:47.526224 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.525814    2582 manager.go:217] Machine: {Timestamp:2026-11-02 13:42:47.524351721 +0000 UTC m=+0.317533546 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101040 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec238c3f97d96f6a70d8ccbed1707f65 SystemUUID:ec238c3f-97d9-6f6a-70d8-ccbed1707f65 BootID:56f3a6b2-2d88-4534-bce7-ba50c0584984 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f3:5b:7b:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f3:5b:7b:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:0c:fd:30:c4:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 13:42:47.526889 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.526868    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 13:42:47.527041 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.527024    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 13:42:47.530669 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.530640    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 13:42:47.530810 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.530669    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 13:42:47.530868 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.530812    2582 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 13:42:47.530868 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.530821    2582 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 13:42:47.530868 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.530857    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:42:47.530942 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.530875    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:42:47.532060 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.532049    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 13:42:47.536619 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.536607    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 13:42:47.536750 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.536741    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 13:42:47.537617 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.537606    2582 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 13:42:47.537653 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.537622    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 13:42:47.537653 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.537633    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 13:42:47.537653 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.537642    2582 kubelet.go:405] "Adding apiserver pod source"
Nov 02 13:42:47.537653 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.537649    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 13:42:47.537987 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.537972    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:42:47.540505 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.540489    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 13:42:47.548030 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.548015    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:42:47.549003 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.548989    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 13:42:47.549083 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549029    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 13:42:47.549224 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549211    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 13:42:47.549281 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549241    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 13:42:47.549281 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549256    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 13:42:47.549281 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549269    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 13:42:47.549281 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549279    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 13:42:47.549391 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549288    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 13:42:47.549965 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549955    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 13:42:47.550013 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549968    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 13:42:47.550013 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549979    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 13:42:47.550013 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.549989    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 13:42:47.550013 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.550002    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 13:42:47.550157 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.550016    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 13:42:47.550810 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.550799    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 13:42:47.550868 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.550814    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 13:42:47.554523 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.554508    2582 server.go:1263] "Started kubelet"
Nov 02 13:42:47.554598 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.554568    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 13:42:47.554710 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.554653    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 13:42:47.554823 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.554749    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 13:42:47.556034 ip-10-0-139-26 systemd[1]: Started Kubernetes Kubelet.
Nov 02 13:42:47.556954 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.556181    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 13:42:47.557928 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.557911    2582 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 13:42:47.561500 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.561476    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 13:42:47.564480 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.564461    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 13:42:47.565145 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.565129    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 13:42:47.565281 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.565260    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:47.565910 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.565893    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 13:42:47.565910 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.565912    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 13:42:47.566057 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.565893    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 13:42:47.566142 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.566127    2582 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 13:42:47.566142 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.566136    2582 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 13:42:47.566662 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.566627    2582 factory.go:153] Registering CRI-O factory
Nov 02 13:42:47.566748 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.566667    2582 factory.go:223] Registration of the crio container factory successfully
Nov 02 13:42:47.566814 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.566753    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 13:42:47.566814 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.566773    2582 factory.go:55] Registering systemd factory
Nov 02 13:42:47.566814 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.566800    2582 factory.go:223] Registration of the systemd container factory successfully
Nov 02 13:42:47.566954 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.566844    2582 factory.go:103] Registering Raw factory
Nov 02 13:42:47.566954 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.566868    2582 manager.go:1199] Started watching for new ooms in manager
Nov 02 13:42:47.569571 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.569554    2582 manager.go:319] Starting recovery of all containers
Nov 02 13:42:47.582972 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.582954    2582 manager.go:324] Recovery completed
Nov 02 13:42:47.587338 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.587322    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:47.589313 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.589289    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:47.589383 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.589329    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:47.589383 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.589343    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:47.589891 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.589876    2582 cpu_manager.go:226] "Starting" policy="none"
Nov 02 13:42:47.589891 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.589890    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 13:42:47.590031 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.589878    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:47.590031 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.589910    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 13:42:47.590549 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.590534    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:47.590615 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.590557    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:47.590615 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.590568    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:47.594071 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.594049    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 13:42:47.594071 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.594052    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-26.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 13:42:47.594564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.594544    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 13:42:47.595110 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.594053    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-26.ec2.internal.18e43e3954a53896  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-26.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-26.ec2.internal,},FirstTimestamp:2026-11-02 13:42:47.554472086 +0000 UTC m=+0.347653908,LastTimestamp:2026-11-02 13:42:47.554472086 +0000 UTC m=+0.347653908,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-26.ec2.internal,}"
Nov 02 13:42:47.640574 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.624891    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tsv5j"
Nov 02 13:42:47.640574 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.632079    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 13:42:47.640574 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.633202    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 13:42:47.640574 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.633218    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 13:42:47.640574 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.633233    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 13:42:47.640574 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.633268    2582 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 13:42:47.640574 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.636884    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tsv5j"
Nov 02 13:42:47.641808 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.641385    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-26.ec2.internal.18e43e3956b8ddcd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-26.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-26.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-26.ec2.internal,},FirstTimestamp:2026-11-02 13:42:47.589313997 +0000 UTC m=+0.382495820,LastTimestamp:2026-11-02 13:42:47.589313997 +0000 UTC m=+0.382495820,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-26.ec2.internal,}"
Nov 02 13:42:47.641909 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.641892    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 13:42:47.654837 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.654818    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-26.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 13:42:47.665684 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.665668    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:47.717955 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.717932    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:47.721003 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.720985    2582 policy_none.go:50] "Start"
Nov 02 13:42:47.721003 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.721002    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 13:42:47.721125 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.721015    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 13:42:47.721125 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.721090    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:47.723811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.723761    2582 policy_none.go:44] "Start"
Nov 02 13:42:47.733948 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.733934    2582 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 13:42:47.765343 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.765308    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 13:42:47.765847 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.765835    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:47.769285 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.769267    2582 manager.go:341] "Starting Device Plugin manager"
Nov 02 13:42:47.769340 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.769290    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 13:42:47.769340 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.769298    2582 server.go:87] "Starting device plugin registration server"
Nov 02 13:42:47.769482 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.769472    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 13:42:47.769537 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.769482    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 13:42:47.769587 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.769566    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 13:42:47.769665 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.769640    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 13:42:47.769665 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.769649    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 13:42:47.770255 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.770236    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 13:42:47.770350 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.770278    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:47.825575 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.825556    2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-26.ec2.internal\" not found" node="ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.870376 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.870360    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:47.871087 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.871073    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:47.871151 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.871101    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:47.871151 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.871111    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:47.871151 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.871133    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.893345 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.893327    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.893408 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.893350    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-26.ec2.internal\": node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:47.934991 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.934969    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"]
Nov 02 13:42:47.935063 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.935017    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:47.936010 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.935997    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:47.936100 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.936025    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:47.936100 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.936039    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:47.938251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.938235    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:47.938401 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.938388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.938461 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.938419    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:47.938974 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.938957    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:47.939044 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.938986    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:47.939044 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.938961    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:47.939044 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.938998    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:47.939044 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.939037    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:47.939178 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.939052    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:47.941030 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.941016    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.941101 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.941040    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:47.941710 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.941692    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:47.941791 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.941718    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:47.941791 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.941728    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:47.946541 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.946525    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:47.967384 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.967287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.967384 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.967326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.967384 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:47.967351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3694267f060f6a7f596609ec14c03ba4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.ec2.internal\" (UID: \"3694267f060f6a7f596609ec14c03ba4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.968238 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.968219    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-26.ec2.internal\" not found" node="ip-10-0-139-26.ec2.internal"
Nov 02 13:42:47.970856 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:47.970841    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-26.ec2.internal\" not found" node="ip-10-0-139-26.ec2.internal"
Nov 02 13:42:48.047043 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.046983    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.068234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.068212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:48.068304 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.068238    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3694267f060f6a7f596609ec14c03ba4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.ec2.internal\" (UID: \"3694267f060f6a7f596609ec14c03ba4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:48.068304 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.068256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:48.068370 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.068305    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:48.068370 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.068320    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3694267f060f6a7f596609ec14c03ba4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.ec2.internal\" (UID: \"3694267f060f6a7f596609ec14c03ba4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:48.068370 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.068320    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:48.147974 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.147950    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.249012 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.248982    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.273405 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.273376    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:48.275560 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.275542    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:48.349502 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.349449    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.450007 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.449984    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.498424 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.498401    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 13:42:48.500216 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.498844    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 13:42:48.500216 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.498514    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 13:42:48.550522 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.550498    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.639910 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.639850    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:48.642114 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.642086    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:37:48 +0000 UTC" deadline="2028-06-28 11:50:43.860995633 +0000 UTC"
Nov 02 13:42:48.642172 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.642116    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14494h7m55.218882903s"
Nov 02 13:42:48.651296 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.651276    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.751928 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.751903    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.834973 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:48.834938    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3a0abeec74791fe4e4176d1d343021.slice/crio-645611df7ab78705ebae02c65af9ccd48c54ce0fe0200f88a4df8134bc435cf8 WatchSource:0}: Error finding container 645611df7ab78705ebae02c65af9ccd48c54ce0fe0200f88a4df8134bc435cf8: Status 404 returned error can't find the container with id 645611df7ab78705ebae02c65af9ccd48c54ce0fe0200f88a4df8134bc435cf8
Nov 02 13:42:48.835666 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:48.835644    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3694267f060f6a7f596609ec14c03ba4.slice/crio-4c54c3e9ab3bcab4bbd3dc7ec62e43b33a3cfa789b1adb1cf482831ce5afa8c6 WatchSource:0}: Error finding container 4c54c3e9ab3bcab4bbd3dc7ec62e43b33a3cfa789b1adb1cf482831ce5afa8c6: Status 404 returned error can't find the container with id 4c54c3e9ab3bcab4bbd3dc7ec62e43b33a3cfa789b1adb1cf482831ce5afa8c6
Nov 02 13:42:48.838913 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.838893    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:42:48.840595 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.840578    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:42:48.840691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.840616    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:42:48.840691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.840629    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:42:48.842622 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.842597    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:42:48.852833 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.852817    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.954342 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:48.954269    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:42:48.977883 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:48.977848    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:49.065674 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.065646    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:49.099867 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.099847    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:42:49.099982 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.099959    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:42:49.122460 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.122437    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:42:49.521089 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.521057    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:49.538450 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.538427    2582 apiserver.go:52] "Watching apiserver"
Nov 02 13:42:49.565404 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.565381    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 13:42:49.573542 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.573518    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 13:42:49.573888 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.573866    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rfknw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal","openshift-network-operator/iptables-alerter-nsnll","kube-system/global-pull-secret-syncer-zp7ql","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv","openshift-cluster-node-tuning-operator/tuned-5tzss","openshift-multus/multus-additional-cni-plugins-vnt96","kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal","openshift-dns/node-resolver-t6vhd","openshift-multus/network-metrics-daemon-g8jrw","openshift-network-diagnostics/network-check-target-2gjsj","openshift-image-registry/node-ca-29jl9","openshift-multus/multus-5sp5j","openshift-ovn-kubernetes/ovnkube-node-7mx2k"]
Nov 02 13:42:49.576775 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.576753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.578980 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.578960    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.581366 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.581347    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.583849 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.583760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.586233 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.586211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.588771 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.588750    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.591826 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.591660    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.595103 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.594658    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:49.596942 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.596667    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 13:42:49.596942 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.596788    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 13:42:49.596942 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.596908    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w4l5x\""
Nov 02 13:42:49.597254 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.597231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:49.597355 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.597306    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:42:49.597719 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.597699    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 13:42:49.598386 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.598368    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4wcws\""
Nov 02 13:42:49.598589 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.598571    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 13:42:49.598589 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.598583    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 13:42:49.598706 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.598630    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 13:42:49.598706 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.598652    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 13:42:49.599408 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.599389    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 13:42:49.599787 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.599770    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 13:42:49.599986 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.599970    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:42:49.600244 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.600227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:49.600476 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.600446    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 13:42:49.600546 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.600527    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qnz8k\""
Nov 02 13:42:49.600675 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.600644    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:42:49.601483 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.601464    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 13:42:49.601761 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.601744    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 13:42:49.602321 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.601851    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 13:42:49.602321 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.601851    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6lw86\""
Nov 02 13:42:49.604953 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.604930    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 13:42:49.605053 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.605007    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bqvhh\""
Nov 02 13:42:49.605053 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.605037    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 13:42:49.605179 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.605072    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 13:42:49.606519 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.606498    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 13:42:49.606650 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.606547    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:42:49.606880 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.606842    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lvdl4\""
Nov 02 13:42:49.607372 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.607354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:49.607467 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.607417    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:42:49.610568 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.610548    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-52vg6\""
Nov 02 13:42:49.610652 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.610626    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 13:42:49.610652 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.610637    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 13:42:49.610733 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.610701    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 13:42:49.610733 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.610706    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 13:42:49.610836 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.610786    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 13:42:49.611422 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.611404    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.613397 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.613370    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 13:42:49.613500 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.613421    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-47zgd\""
Nov 02 13:42:49.613500 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.613470    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 13:42:49.622718 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.622691    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 13:42:49.622822 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.622756    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dsgm8"
Nov 02 13:42:49.633247 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.633227    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dsgm8"
Nov 02 13:42:49.638722 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.638676    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal" event={"ID":"3694267f060f6a7f596609ec14c03ba4","Type":"ContainerStarted","Data":"4c54c3e9ab3bcab4bbd3dc7ec62e43b33a3cfa789b1adb1cf482831ce5afa8c6"}
Nov 02 13:42:49.640754 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.640731    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal" event={"ID":"7b3a0abeec74791fe4e4176d1d343021","Type":"ContainerStarted","Data":"645611df7ab78705ebae02c65af9ccd48c54ce0fe0200f88a4df8134bc435cf8"}
Nov 02 13:42:49.667333 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.667317    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 13:42:49.679177 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4fcq\" (UniqueName: \"kubernetes.io/projected/3b44af93-ab31-483d-9eab-3e996c90eee7-kube-api-access-d4fcq\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.679177 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d26dd5a-118a-4d50-9661-22feeb24d158-konnectivity-ca\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:49.679177 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679065    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrph\" (UniqueName: \"kubernetes.io/projected/132bf469-fbc8-419c-b19a-17bcabfecce1-kube-api-access-hsrph\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.679177 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-node-log\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.679177 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-run\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-sys\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679237    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91358478-4176-40c6-85b3-9e044fd2fa22-iptables-alerter-script\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-netns\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-etc-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8dr7\" (UniqueName: \"kubernetes.io/projected/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-kube-api-access-n8dr7\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679400    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-systemd\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-slash\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-conf-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91358478-4176-40c6-85b3-9e044fd2fa22-host-slash\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.679510 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679504    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-system-cni-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f2db88-b241-451f-8f1e-9f8c68ad0e73-tmp-dir\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-env-overrides\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-modprobe-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-netns\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-var-lib-kubelet\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ktl\" (UniqueName: \"kubernetes.io/projected/4366155b-b272-4b13-b328-4d9f87325b84-kube-api-access-98ktl\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-socket-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-ovn\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-kubernetes\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-tmp\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-socket-dir-parent\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-kubelet\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.680018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-cnibin\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d26dd5a-118a-4d50-9661-22feeb24d158-agent-certs\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.679992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rlg\" (UniqueName: \"kubernetes.io/projected/91600010-0a54-4517-b9a7-1cc38b075abd-kube-api-access-47rlg\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-serviceca\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680136    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-conf\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9f7\" (UniqueName: \"kubernetes.io/projected/91358478-4176-40c6-85b3-9e044fd2fa22-kube-api-access-cc9f7\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-bin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-etc-kubernetes\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-etc-tuned\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-kubelet-config\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjt9s\" (UniqueName: \"kubernetes.io/projected/26f2db88-b241-451f-8f1e-9f8c68ad0e73-kube-api-access-sjt9s\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-systemd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680412    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-kubelet\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.680564 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-config\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovn-node-metrics-cert\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680504    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-host\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-dbus\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-bin\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-netd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cnibin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-hostroot\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-multus-certs\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680738    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v5tx\" (UniqueName: \"kubernetes.io/projected/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-kube-api-access-4v5tx\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680837    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-device-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.680897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bx75\" (UniqueName: \"kubernetes.io/projected/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-kube-api-access-2bx75\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-registration-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.681168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-daemon-config\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681196    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-sys-fs\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-log-socket\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-system-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f2db88-b241-451f-8f1e-9f8c68ad0e73-hosts-file\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-systemd-units\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-var-lib-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cni-binary-copy\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681425    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-script-lib\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-host\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-lib-modules\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681650    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-k8s-cni-cncf-io\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.681691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-multus\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.682224 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681687    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.682224 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.681703    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysconfig\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.705478 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.705439    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 13:42:49.705572 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.705503    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5dqzs\""
Nov 02 13:42:49.782468 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4fcq\" (UniqueName: \"kubernetes.io/projected/3b44af93-ab31-483d-9eab-3e996c90eee7-kube-api-access-d4fcq\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.782594 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d26dd5a-118a-4d50-9661-22feeb24d158-konnectivity-ca\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:49.782594 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrph\" (UniqueName: \"kubernetes.io/projected/132bf469-fbc8-419c-b19a-17bcabfecce1-kube-api-access-hsrph\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.782594 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-node-log\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.782762 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-run\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.782762 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-sys\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.782762 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91358478-4176-40c6-85b3-9e044fd2fa22-iptables-alerter-script\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.782762 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-netns\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.782762 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782702    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-etc-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.782762 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782719    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dr7\" (UniqueName: \"kubernetes.io/projected/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-kube-api-access-n8dr7\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.782762 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-systemd\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.782762 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782750    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-slash\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.782762 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-conf-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91358478-4176-40c6-85b3-9e044fd2fa22-host-slash\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782827    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-run\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782818    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-system-cni-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782897    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f2db88-b241-451f-8f1e-9f8c68ad0e73-tmp-dir\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-env-overrides\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-modprobe-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782956    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.782995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-netns\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783011    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-var-lib-kubelet\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783042    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-node-log\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783060    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-var-lib-kubelet\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ktl\" (UniqueName: \"kubernetes.io/projected/4366155b-b272-4b13-b328-4d9f87325b84-kube-api-access-98ktl\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783113    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-socket-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-ovn\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.783249 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-kubernetes\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-tmp\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783164    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d26dd5a-118a-4d50-9661-22feeb24d158-konnectivity-ca\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783179    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-socket-dir-parent\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783197    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-kubelet\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-cnibin\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783261    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-cnibin\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783284    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d26dd5a-118a-4d50-9661-22feeb24d158-agent-certs\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783291    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-sys\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rlg\" (UniqueName: \"kubernetes.io/projected/91600010-0a54-4517-b9a7-1cc38b075abd-kube-api-access-47rlg\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-serviceca\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783335    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-conf\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9f7\" (UniqueName: \"kubernetes.io/projected/91358478-4176-40c6-85b3-9e044fd2fa22-kube-api-access-cc9f7\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-bin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-etc-kubernetes\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.783977 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-etc-tuned\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-kubelet-config\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjt9s\" (UniqueName: \"kubernetes.io/projected/26f2db88-b241-451f-8f1e-9f8c68ad0e73-kube-api-access-sjt9s\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783493    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-systemd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-kubelet\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-config\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783570    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-slash\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783607    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-system-cni-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovn-node-metrics-cert\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-host\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783653    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-dbus\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783613    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91358478-4176-40c6-85b3-9e044fd2fa22-host-slash\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783567    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-netns\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783714    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-conf-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.784811 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783742    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-host\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783746    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-bin\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783738    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-etc-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783783    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-netd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783827    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-bin\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783834    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-dbus\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cnibin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783710    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-hostroot\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-multus-certs\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783986    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-hostroot\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783993    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-modprobe-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783993    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-kubelet\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5tx\" (UniqueName: \"kubernetes.io/projected/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-kube-api-access-4v5tx\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.783541    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-systemd\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-device-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784056    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784059    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.785712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784131    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-multus-certs\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784166    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-netd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784210    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784223    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784259    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-systemd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784373    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-bin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784454    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-etc-kubernetes\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784482    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-serviceca\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784534    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cnibin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784535    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784548    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-kubelet-config\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784609    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-device-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bx75\" (UniqueName: \"kubernetes.io/projected/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-kube-api-access-2bx75\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-registration-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-daemon-config\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-sys-fs\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784748    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-kubernetes\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.786625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-log-socket\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784761    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-kubelet\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784729    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-conf\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784947    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-socket-dir-parent\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.784990    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-netns\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785073    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-socket-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785093    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-sys-fs\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785104    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-registration-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785134    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-ovn\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-system-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f2db88-b241-451f-8f1e-9f8c68ad0e73-hosts-file\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785222    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f2db88-b241-451f-8f1e-9f8c68ad0e73-tmp-dir\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-systemd-units\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-var-lib-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785385    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cni-binary-copy\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785459    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-system-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.787458 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785489    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91358478-4176-40c6-85b3-9e044fd2fa22-iptables-alerter-script\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785515    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-systemd-units\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785512    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-script-lib\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785546    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-var-lib-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785549    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785488    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-log-socket\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785417    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-host\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785549    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f2db88-b241-451f-8f1e-9f8c68ad0e73-hosts-file\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-lib-modules\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-k8s-cni-cncf-io\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785931    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-multus\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.785988    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-daemon-config\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysconfig\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.786035    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:49.788371 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786070    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-env-overrides\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786094    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysconfig\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.786129    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret podName:00be7ba2-6f06-4ed5-a007-d0ac612177ee nodeName:}" failed. No retries permitted until 2026-11-02 13:42:50.286094816 +0000 UTC m=+3.079276642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret") pod "global-pull-secret-syncer-zp7ql" (UID: "00be7ba2-6f06-4ed5-a007-d0ac612177ee") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786135    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-host\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786246    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-config\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786355    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-k8s-cni-cncf-io\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.786443    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786444    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.786516    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:42:50.286473084 +0000 UTC m=+3.079654912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786591    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-lib-modules\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786898    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-multus\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.786965    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.787026    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.787151    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.789124 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.787962    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cni-binary-copy\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.790111 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.790090    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-tmp\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.790229 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.790210    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovn-node-metrics-cert\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.791018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.790998    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-etc-tuned\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.791449 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.791425    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-script-lib\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.791768 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.791732    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d26dd5a-118a-4d50-9661-22feeb24d158-agent-certs\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:49.813876 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.810627    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4fcq\" (UniqueName: \"kubernetes.io/projected/3b44af93-ab31-483d-9eab-3e996c90eee7-kube-api-access-d4fcq\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.813876 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.812790    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjt9s\" (UniqueName: \"kubernetes.io/projected/26f2db88-b241-451f-8f1e-9f8c68ad0e73-kube-api-access-sjt9s\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.814340 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.814316    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ktl\" (UniqueName: \"kubernetes.io/projected/4366155b-b272-4b13-b328-4d9f87325b84-kube-api-access-98ktl\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.814470 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.814446    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dr7\" (UniqueName: \"kubernetes.io/projected/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-kube-api-access-n8dr7\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.816183 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.815793    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rlg\" (UniqueName: \"kubernetes.io/projected/91600010-0a54-4517-b9a7-1cc38b075abd-kube-api-access-47rlg\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:49.816183 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.815988    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bx75\" (UniqueName: \"kubernetes.io/projected/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-kube-api-access-2bx75\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.816183 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.816146    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9f7\" (UniqueName: \"kubernetes.io/projected/91358478-4176-40c6-85b3-9e044fd2fa22-kube-api-access-cc9f7\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.816355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.816189    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5tx\" (UniqueName: \"kubernetes.io/projected/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-kube-api-access-4v5tx\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.817079 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.817058    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:49.817079 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.817078    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:49.817199 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.817092    2582 projected.go:196] Error preparing data for projected volume kube-api-access-47vrz for pod openshift-network-diagnostics/network-check-target-2gjsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:49.817199 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:49.817138    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz podName:fb318d8f-e7de-4185-96cd-d592b304248a nodeName:}" failed. No retries permitted until 2026-11-02 13:42:50.317125096 +0000 UTC m=+3.110306920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-47vrz" (UniqueName: "kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz") pod "network-check-target-2gjsj" (UID: "fb318d8f-e7de-4185-96cd-d592b304248a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:49.818338 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.818306    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrph\" (UniqueName: \"kubernetes.io/projected/132bf469-fbc8-419c-b19a-17bcabfecce1-kube-api-access-hsrph\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.892604 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.892577    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:42:49.903426 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:49.903382    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f2db88_b241_451f_8f1e_9f8c68ad0e73.slice/crio-bd1fe85b457b64df005e91ff653c06eb95b1a5477175808c82adb84758a4e403 WatchSource:0}: Error finding container bd1fe85b457b64df005e91ff653c06eb95b1a5477175808c82adb84758a4e403: Status 404 returned error can't find the container with id bd1fe85b457b64df005e91ff653c06eb95b1a5477175808c82adb84758a4e403
Nov 02 13:42:49.918221 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.918190    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:42:49.925644 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.925398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:42:49.925644 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.925599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:42:49.931703 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:49.930716    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91358478_4176_40c6_85b3_9e044fd2fa22.slice/crio-30179d4d8ff0b96d14fec5a86f48c3a5b04e6b73264111ecab3a796544789e4c WatchSource:0}: Error finding container 30179d4d8ff0b96d14fec5a86f48c3a5b04e6b73264111ecab3a796544789e4c: Status 404 returned error can't find the container with id 30179d4d8ff0b96d14fec5a86f48c3a5b04e6b73264111ecab3a796544789e4c
Nov 02 13:42:49.935659 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.935565    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:42:49.940125 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.939952    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:42:49.942867 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:49.942839    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b5b1d2_b605_4f94_aad7_c7f01d84d6bd.slice/crio-177cfdfed6d877425545577f69af401ace0cb6d7f42a3976a12aca437670a2ef WatchSource:0}: Error finding container 177cfdfed6d877425545577f69af401ace0cb6d7f42a3976a12aca437670a2ef: Status 404 returned error can't find the container with id 177cfdfed6d877425545577f69af401ace0cb6d7f42a3976a12aca437670a2ef
Nov 02 13:42:49.946084 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:49.946037    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46f80e0_332a_4c19_81a2_08cbb65f7fb8.slice/crio-510c37485b484c9d4da49a3a5cbbd610d442bfe6f240ca2197019c0752c7b1e0 WatchSource:0}: Error finding container 510c37485b484c9d4da49a3a5cbbd610d442bfe6f240ca2197019c0752c7b1e0: Status 404 returned error can't find the container with id 510c37485b484c9d4da49a3a5cbbd610d442bfe6f240ca2197019c0752c7b1e0
Nov 02 13:42:49.947798 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:49.947769    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b44af93_ab31_483d_9eab_3e996c90eee7.slice/crio-81e9e260477d75bf602d68d90ce95eff15df2c45c8da9ce1069ab7c2b66778b1 WatchSource:0}: Error finding container 81e9e260477d75bf602d68d90ce95eff15df2c45c8da9ce1069ab7c2b66778b1: Status 404 returned error can't find the container with id 81e9e260477d75bf602d68d90ce95eff15df2c45c8da9ce1069ab7c2b66778b1
Nov 02 13:42:49.961315 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.961288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:42:49.965548 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.965400    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:49.967872 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:49.967851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5sp5j"
Nov 02 13:42:49.972735 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:49.972709    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132bf469_fbc8_419c_b19a_17bcabfecce1.slice/crio-5e53990fbb91dd58e726bcba38adb28b827ed817e5b7357a968b5b7b6a1fab76 WatchSource:0}: Error finding container 5e53990fbb91dd58e726bcba38adb28b827ed817e5b7357a968b5b7b6a1fab76: Status 404 returned error can't find the container with id 5e53990fbb91dd58e726bcba38adb28b827ed817e5b7357a968b5b7b6a1fab76
Nov 02 13:42:49.975239 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:49.975204    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d26dd5a_118a_4d50_9661_22feeb24d158.slice/crio-fc871305024afeb57f42699ecdccdfea0dd868ba79ebbcaed9d32ccc514caff8 WatchSource:0}: Error finding container fc871305024afeb57f42699ecdccdfea0dd868ba79ebbcaed9d32ccc514caff8: Status 404 returned error can't find the container with id fc871305024afeb57f42699ecdccdfea0dd868ba79ebbcaed9d32ccc514caff8
Nov 02 13:42:49.981643 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:42:49.981619    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d67e5a_851c_4fdd_befa_c0a97e93d5d2.slice/crio-30b87854575dd5d5baf607c9db02cd611b7f81465fe52a5b8ba15cbf5328b072 WatchSource:0}: Error finding container 30b87854575dd5d5baf607c9db02cd611b7f81465fe52a5b8ba15cbf5328b072: Status 404 returned error can't find the container with id 30b87854575dd5d5baf607c9db02cd611b7f81465fe52a5b8ba15cbf5328b072
Nov 02 13:42:50.018959 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.018929    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:42:50.291111 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.291081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:50.291271 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.291123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:50.291271 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:50.291218    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:50.291271 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:50.291223    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:50.291385 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:50.291276    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret podName:00be7ba2-6f06-4ed5-a007-d0ac612177ee nodeName:}" failed. No retries permitted until 2026-11-02 13:42:51.291260143 +0000 UTC m=+4.084441956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret") pod "global-pull-secret-syncer-zp7ql" (UID: "00be7ba2-6f06-4ed5-a007-d0ac612177ee") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:50.291385 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:50.291292    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:42:51.29128507 +0000 UTC m=+4.084466878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:50.392378 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.392346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:50.392538 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:50.392517    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:50.392610 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:50.392548    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:50.392610 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:50.392562    2582 projected.go:196] Error preparing data for projected volume kube-api-access-47vrz for pod openshift-network-diagnostics/network-check-target-2gjsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:50.392715 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:50.392620    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz podName:fb318d8f-e7de-4185-96cd-d592b304248a nodeName:}" failed. No retries permitted until 2026-11-02 13:42:51.392599457 +0000 UTC m=+4.185781271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-47vrz" (UniqueName: "kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz") pod "network-check-target-2gjsj" (UID: "fb318d8f-e7de-4185-96cd-d592b304248a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:50.634363 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.633750    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:50.634363 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:50.633905    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:42:50.636601 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.636503    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:37:50 +0000 UTC" deadline="2028-07-11 06:19:34.91293831 +0000 UTC"
Nov 02 13:42:50.636601 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.636537    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14800h36m44.276406324s"
Nov 02 13:42:50.646647 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.646616    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t6vhd" event={"ID":"26f2db88-b241-451f-8f1e-9f8c68ad0e73","Type":"ContainerStarted","Data":"bd1fe85b457b64df005e91ff653c06eb95b1a5477175808c82adb84758a4e403"}
Nov 02 13:42:50.670592 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.670508    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnt96" event={"ID":"132bf469-fbc8-419c-b19a-17bcabfecce1","Type":"ContainerStarted","Data":"5e53990fbb91dd58e726bcba38adb28b827ed817e5b7357a968b5b7b6a1fab76"}
Nov 02 13:42:50.675223 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.675149    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv" event={"ID":"3b44af93-ab31-483d-9eab-3e996c90eee7","Type":"ContainerStarted","Data":"81e9e260477d75bf602d68d90ce95eff15df2c45c8da9ce1069ab7c2b66778b1"}
Nov 02 13:42:50.677276 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.677202    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-29jl9" event={"ID":"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd","Type":"ContainerStarted","Data":"177cfdfed6d877425545577f69af401ace0cb6d7f42a3976a12aca437670a2ef"}
Nov 02 13:42:50.683781 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.683674    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" event={"ID":"f46f80e0-332a-4c19-81a2-08cbb65f7fb8","Type":"ContainerStarted","Data":"510c37485b484c9d4da49a3a5cbbd610d442bfe6f240ca2197019c0752c7b1e0"}
Nov 02 13:42:50.692234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.692209    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5sp5j" event={"ID":"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2","Type":"ContainerStarted","Data":"30b87854575dd5d5baf607c9db02cd611b7f81465fe52a5b8ba15cbf5328b072"}
Nov 02 13:42:50.695775 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.695748    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rfknw" event={"ID":"5d26dd5a-118a-4d50-9661-22feeb24d158","Type":"ContainerStarted","Data":"fc871305024afeb57f42699ecdccdfea0dd868ba79ebbcaed9d32ccc514caff8"}
Nov 02 13:42:50.698531 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.698471    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5tzss" event={"ID":"4366155b-b272-4b13-b328-4d9f87325b84","Type":"ContainerStarted","Data":"42f3f887256ace329440381ef53613d7587948976676ae9272dc54be82c1f246"}
Nov 02 13:42:50.704538 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:50.704506    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nsnll" event={"ID":"91358478-4176-40c6-85b3-9e044fd2fa22","Type":"ContainerStarted","Data":"30179d4d8ff0b96d14fec5a86f48c3a5b04e6b73264111ecab3a796544789e4c"}
Nov 02 13:42:51.301183 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.301150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:51.301323 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.301214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:51.301383 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.301371    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:51.301446 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.301427    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret podName:00be7ba2-6f06-4ed5-a007-d0ac612177ee nodeName:}" failed. No retries permitted until 2026-11-02 13:42:53.301409648 +0000 UTC m=+6.094591480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret") pod "global-pull-secret-syncer-zp7ql" (UID: "00be7ba2-6f06-4ed5-a007-d0ac612177ee") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:51.301519 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.301493    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:51.301571 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.301526    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:42:53.301514507 +0000 UTC m=+6.094696330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:51.402291 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.402254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:51.402462 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.402430    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:51.402462 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.402451    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:51.402578 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.402465    2582 projected.go:196] Error preparing data for projected volume kube-api-access-47vrz for pod openshift-network-diagnostics/network-check-target-2gjsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:51.402578 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.402521    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz podName:fb318d8f-e7de-4185-96cd-d592b304248a nodeName:}" failed. No retries permitted until 2026-11-02 13:42:53.40250223 +0000 UTC m=+6.195684057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-47vrz" (UniqueName: "kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz") pod "network-check-target-2gjsj" (UID: "fb318d8f-e7de-4185-96cd-d592b304248a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:51.641693 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.641659    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:51.642259 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.641811    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:42:51.642259 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.642233    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:51.642393 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:51.642329    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:42:51.642550 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.642417    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:37:50 +0000 UTC" deadline="2028-07-04 14:00:29.406539906 +0000 UTC"
Nov 02 13:42:51.642550 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.642461    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14640h17m37.764082581s"
Nov 02 13:42:51.717710 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.717661    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal" event={"ID":"3694267f060f6a7f596609ec14c03ba4","Type":"ContainerStarted","Data":"b618146706c124ded121af7110a1533e01f647d5af2bb3d2711c3fc9d76d06f1"}
Nov 02 13:42:51.725326 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.725293    2582 generic.go:356] "Generic (PLEG): container finished" podID="7b3a0abeec74791fe4e4176d1d343021" containerID="d1afcc17a1ae9aa3ed4d91174327d389b70ebd9f1f2d8e9131b3f74d7433a8a3" exitCode=0
Nov 02 13:42:51.725478 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.725344    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal" event={"ID":"7b3a0abeec74791fe4e4176d1d343021","Type":"ContainerDied","Data":"d1afcc17a1ae9aa3ed4d91174327d389b70ebd9f1f2d8e9131b3f74d7433a8a3"}
Nov 02 13:42:51.766939 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:51.766145    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal" podStartSLOduration=2.766127343 podStartE2EDuration="2.766127343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:51.765737479 +0000 UTC m=+4.558919313" watchObservedRunningTime="2026-11-02 13:42:51.766127343 +0000 UTC m=+4.559309176"
Nov 02 13:42:52.634705 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:52.634163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:52.634705 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:52.634318    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:42:52.736954 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:52.736875    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal" event={"ID":"7b3a0abeec74791fe4e4176d1d343021","Type":"ContainerStarted","Data":"7ec0fc4f93068ed62ec3fdb413bfb26ba9d3919f4ddaa55138e3260ac153fa4d"}
Nov 02 13:42:53.320097 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:53.320058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:53.320261 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:53.320128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:53.320317 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.320288    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:53.320388 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.320352    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret podName:00be7ba2-6f06-4ed5-a007-d0ac612177ee nodeName:}" failed. No retries permitted until 2026-11-02 13:42:57.320333203 +0000 UTC m=+10.113515027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret") pod "global-pull-secret-syncer-zp7ql" (UID: "00be7ba2-6f06-4ed5-a007-d0ac612177ee") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:53.320784 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.320764    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:53.320900 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.320823    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:42:57.320805809 +0000 UTC m=+10.113987637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:53.421181 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:53.421141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:53.421360 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.421345    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:53.421431 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.421371    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:53.421431 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.421385    2582 projected.go:196] Error preparing data for projected volume kube-api-access-47vrz for pod openshift-network-diagnostics/network-check-target-2gjsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:53.421532 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.421443    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz podName:fb318d8f-e7de-4185-96cd-d592b304248a nodeName:}" failed. No retries permitted until 2026-11-02 13:42:57.421424852 +0000 UTC m=+10.214606674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-47vrz" (UniqueName: "kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz") pod "network-check-target-2gjsj" (UID: "fb318d8f-e7de-4185-96cd-d592b304248a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:53.636623 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:53.636590    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:53.636834 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.636733    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:42:53.637168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:53.637148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:53.637290 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:53.637268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:42:53.750512 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:53.750474    2582 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="3b02d5b1c144cde1979c95160b5c2eb21c420f240a08e3f98346023aea532bf0" exitCode=0
Nov 02 13:42:53.751009 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:53.750722    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnt96" event={"ID":"132bf469-fbc8-419c-b19a-17bcabfecce1","Type":"ContainerDied","Data":"3b02d5b1c144cde1979c95160b5c2eb21c420f240a08e3f98346023aea532bf0"}
Nov 02 13:42:53.805481 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:53.805418    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal" podStartSLOduration=4.805397703 podStartE2EDuration="4.805397703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:52.785498363 +0000 UTC m=+5.578680194" watchObservedRunningTime="2026-11-02 13:42:53.805397703 +0000 UTC m=+6.598579537"
Nov 02 13:42:54.633737 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:54.633704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:54.633950 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:54.633856    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:42:54.764516 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:54.764425    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rfknw" event={"ID":"5d26dd5a-118a-4d50-9661-22feeb24d158","Type":"ContainerStarted","Data":"d85f91a6098fcb2f96d2a96f086aad2f3c84a62645244b0dd16cc6333ce85ca6"}
Nov 02 13:42:55.639057 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:55.638258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:55.639057 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:55.638398    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:42:55.639057 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:55.638831    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:55.639057 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:55.638953    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:42:55.771298 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:55.771260    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-29jl9" event={"ID":"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd","Type":"ContainerStarted","Data":"c44c2969f2bac24b2ea0bd27ac2c29183c80eb048b689b4417336053a0fbde33"}
Nov 02 13:42:55.817821 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:55.817759    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rfknw" podStartSLOduration=2.5452447190000003 podStartE2EDuration="6.817738104s" totalImagesPullingTime="4.272493385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:54.813620519 +0000 UTC m=+7.606802355" watchObservedRunningTime="2026-11-02 13:42:55.817738104 +0000 UTC m=+8.610919939"
Nov 02 13:42:56.634628 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:56.634110    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:56.634628 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:56.634251    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:42:56.775644 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:56.775609    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv" event={"ID":"3b44af93-ab31-483d-9eab-3e996c90eee7","Type":"ContainerStarted","Data":"08eeec4c1cbe50e477dd5f2f93829c6d2251897ce570bf6094daec62d8a81407"}
Nov 02 13:42:57.351002 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:57.350471    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:57.351002 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:57.350537    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:57.351002 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.350706    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:57.351002 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.350772    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret podName:00be7ba2-6f06-4ed5-a007-d0ac612177ee nodeName:}" failed. No retries permitted until 2026-11-02 13:43:05.350751351 +0000 UTC m=+18.143933177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret") pod "global-pull-secret-syncer-zp7ql" (UID: "00be7ba2-6f06-4ed5-a007-d0ac612177ee") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:42:57.351002 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.350912    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:57.351002 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.350954    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:43:05.350938388 +0000 UTC m=+18.144120212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:42:57.452109 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:57.451430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:57.452109 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.451613    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:42:57.452109 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.451633    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:42:57.452109 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.451646    2582 projected.go:196] Error preparing data for projected volume kube-api-access-47vrz for pod openshift-network-diagnostics/network-check-target-2gjsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:57.452109 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.451711    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz podName:fb318d8f-e7de-4185-96cd-d592b304248a nodeName:}" failed. No retries permitted until 2026-11-02 13:43:05.451689741 +0000 UTC m=+18.244871575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-47vrz" (UniqueName: "kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz") pod "network-check-target-2gjsj" (UID: "fb318d8f-e7de-4185-96cd-d592b304248a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:42:57.637839 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:57.637745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:57.638012 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:57.637937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:57.638072 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.638041    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:42:57.638632 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:57.638585    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:42:57.652947 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:57.652859    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:57.653573 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:57.653370    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:42:57.704130 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:57.703208    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-29jl9" podStartSLOduration=4.412646185 podStartE2EDuration="9.703187232s" totalImagesPullingTime="5.290541047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:55.81881517 +0000 UTC m=+8.611997005" watchObservedRunningTime="2026-11-02 13:42:57.703187232 +0000 UTC m=+10.496369081"
Nov 02 13:42:57.928744 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:57.928691    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 13:42:58.635081 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.634305    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:42:58.635081 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:58.634649    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:42:58.774935 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.774839    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:42:57.928711622Z","UUID":"de3556c0-e7cb-43da-a2b9-c1c86a9e06e4","Handler":null,"Name":"","Endpoint":""}
Nov 02 13:42:58.776590 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.776563    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 13:42:58.776691 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.776601    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 13:42:58.781612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.781585    2582 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="9798ec8ed1832291b9150443014207dcf7b91d7270fcbda38dc30b049c961ee7" exitCode=0
Nov 02 13:42:58.782062 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.781895    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnt96" event={"ID":"132bf469-fbc8-419c-b19a-17bcabfecce1","Type":"ContainerDied","Data":"9798ec8ed1832291b9150443014207dcf7b91d7270fcbda38dc30b049c961ee7"}
Nov 02 13:42:58.786298 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.786274    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv" event={"ID":"3b44af93-ab31-483d-9eab-3e996c90eee7","Type":"ContainerStarted","Data":"1f9a905d7b6d2652efdf3349bd6b1a01bc6e38a2e58182e82849dbd057c36431"}
Nov 02 13:42:58.789893 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.789870    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5sp5j" event={"ID":"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2","Type":"ContainerStarted","Data":"26e65dc7fd6540e0367633b3d83cbb15c12dd57c1cbdde8319448a6990a96905"}
Nov 02 13:42:58.799360 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.799334    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t6vhd" event={"ID":"26f2db88-b241-451f-8f1e-9f8c68ad0e73","Type":"ContainerStarted","Data":"9ba962d2894d2c2579ec0f3cdf2c195405cbec01d4576e9c5b676024003e8022"}
Nov 02 13:42:58.937636 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:58.937530    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5sp5j" podStartSLOduration=1.590472407 podStartE2EDuration="9.937511115s" totalImagesPullingTime="8.347038708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:58.890398652 +0000 UTC m=+11.683580485" watchObservedRunningTime="2026-11-02 13:42:58.937511115 +0000 UTC m=+11.730692949"
Nov 02 13:42:59.634798 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:59.634197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:42:59.634798 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:59.634242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:42:59.634798 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:59.634343    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:42:59.634798 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:42:59.634739    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:42:59.803053 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:59.803019    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nsnll" event={"ID":"91358478-4176-40c6-85b3-9e044fd2fa22","Type":"ContainerStarted","Data":"54c61c3dbe6d9f939bf3ce2793570a0600a1ce5c294ba9c8e246e278f1897d42"}
Nov 02 13:42:59.805574 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:59.805551    2582 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="9530c571bcd9d358fdc48a4619db4282f4582c0c93e4797b8a9a83c8b7b3e7bd" exitCode=0
Nov 02 13:42:59.805671 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:59.805619    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnt96" event={"ID":"132bf469-fbc8-419c-b19a-17bcabfecce1","Type":"ContainerDied","Data":"9530c571bcd9d358fdc48a4619db4282f4582c0c93e4797b8a9a83c8b7b3e7bd"}
Nov 02 13:42:59.807992 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:59.807955    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv" event={"ID":"3b44af93-ab31-483d-9eab-3e996c90eee7","Type":"ContainerStarted","Data":"71cd3d3c8c45f6a45617718a7024351240b6a3d4f9d42499e9f31dc9383be457"}
Nov 02 13:42:59.877403 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:59.877349    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nsnll" podStartSLOduration=3.484517488 podStartE2EDuration="11.877337686s" totalImagesPullingTime="8.392820198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:59.877116132 +0000 UTC m=+12.670297959" watchObservedRunningTime="2026-11-02 13:42:59.877337686 +0000 UTC m=+12.670519532"
Nov 02 13:42:59.877663 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:59.877637    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-t6vhd" podStartSLOduration=3.217372221 podStartE2EDuration="11.87763125s" totalImagesPullingTime="8.660259029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:58.937417539 +0000 UTC m=+11.730599372" watchObservedRunningTime="2026-11-02 13:42:59.87763125 +0000 UTC m=+12.670813077"
Nov 02 13:42:59.984624 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:42:59.984567    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv" podStartSLOduration=3.346516185 podStartE2EDuration="11.984551447s" totalImagesPullingTime="8.638035262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:59.984399956 +0000 UTC m=+12.777581786" watchObservedRunningTime="2026-11-02 13:42:59.984551447 +0000 UTC m=+12.777733279"
Nov 02 13:43:00.633996 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:00.633968    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:00.634121 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:00.634094    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:00.811707 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:00.811677    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5tzss" event={"ID":"4366155b-b272-4b13-b328-4d9f87325b84","Type":"ContainerStarted","Data":"01d4d82cc79c6cecdae330ecd071e3bc98638d918af12ac6ca04f95c8e1f6552"}
Nov 02 13:43:00.858241 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:00.858200    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5tzss" podStartSLOduration=2.181196208 podStartE2EDuration="12.858188319s" totalImagesPullingTime="10.676992111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:00.858057382 +0000 UTC m=+13.651239213" watchObservedRunningTime="2026-11-02 13:43:00.858188319 +0000 UTC m=+13.651370146"
Nov 02 13:43:01.633640 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:01.633609    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:01.634343 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:01.633736    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:01.634343 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:01.633801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:01.634343 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:01.633890    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:01.816562 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:01.816528    2582 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="8ba17be7daaf2c9c44784d0975b36b82e68d539d8a4cec5d6927e90bdb1ef82b" exitCode=0
Nov 02 13:43:01.816734 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:01.816618    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnt96" event={"ID":"132bf469-fbc8-419c-b19a-17bcabfecce1","Type":"ContainerDied","Data":"8ba17be7daaf2c9c44784d0975b36b82e68d539d8a4cec5d6927e90bdb1ef82b"}
Nov 02 13:43:02.634259 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:02.634228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:02.634722 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:02.634357    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:03.634039 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:03.633851    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:03.634168 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:03.633922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:03.634221 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:03.634180    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:03.634221 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:03.634208    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:03.825130 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:03.825098    2582 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="e2d08a9a6b1c69080764d8a10d0b9198b083ab40cbc69cb3bffc6067f7a48715" exitCode=0
Nov 02 13:43:03.825577 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:03.825159    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnt96" event={"ID":"132bf469-fbc8-419c-b19a-17bcabfecce1","Type":"ContainerDied","Data":"e2d08a9a6b1c69080764d8a10d0b9198b083ab40cbc69cb3bffc6067f7a48715"}
Nov 02 13:43:04.633796 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:04.633754    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:04.634017 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:04.633893    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:04.830466 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:04.830430    2582 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="b409fb81a8cbb74b41f70d37cc361e8af6efa44e5b97b540271bbfdf2d47c8ac" exitCode=0
Nov 02 13:43:04.830958 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:04.830481    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnt96" event={"ID":"132bf469-fbc8-419c-b19a-17bcabfecce1","Type":"ContainerDied","Data":"b409fb81a8cbb74b41f70d37cc361e8af6efa44e5b97b540271bbfdf2d47c8ac"}
Nov 02 13:43:05.416257 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.416229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:05.416444 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.416266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:05.416444 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.416376    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:05.416444 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.416425    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret podName:00be7ba2-6f06-4ed5-a007-d0ac612177ee nodeName:}" failed. No retries permitted until 2026-11-02 13:43:21.416413682 +0000 UTC m=+34.209595489 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret") pod "global-pull-secret-syncer-zp7ql" (UID: "00be7ba2-6f06-4ed5-a007-d0ac612177ee") : object "kube-system"/"original-pull-secret" not registered
Nov 02 13:43:05.416550 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.416376    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:05.416550 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.416498    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:43:21.416486632 +0000 UTC m=+34.209668452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 13:43:05.517390 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.517366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:05.517501 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.517476    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 13:43:05.517501 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.517490    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 13:43:05.517501 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.517498    2582 projected.go:196] Error preparing data for projected volume kube-api-access-47vrz for pod openshift-network-diagnostics/network-check-target-2gjsj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:05.517596 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.517537    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz podName:fb318d8f-e7de-4185-96cd-d592b304248a nodeName:}" failed. No retries permitted until 2026-11-02 13:43:21.517525113 +0000 UTC m=+34.310706932 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-47vrz" (UniqueName: "kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz") pod "network-check-target-2gjsj" (UID: "fb318d8f-e7de-4185-96cd-d592b304248a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 13:43:05.633548 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.633527    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:05.633662 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.633642    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:05.633706 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.633695    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:05.633813 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:05.633797    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:05.835375 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.835175    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vnt96" event={"ID":"132bf469-fbc8-419c-b19a-17bcabfecce1","Type":"ContainerStarted","Data":"293e06a8215c86eb8b1ad6001a494b335cd68ba93f065592c81f25abf8b15d48"}
Nov 02 13:43:05.837703 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.837680    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" event={"ID":"f46f80e0-332a-4c19-81a2-08cbb65f7fb8","Type":"ContainerStarted","Data":"fd8a59ff429f59d08a873197855b8d94af29ec39e5daccad30acc22ce6cdc49d"}
Nov 02 13:43:05.837824 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.837753    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" event={"ID":"f46f80e0-332a-4c19-81a2-08cbb65f7fb8","Type":"ContainerStarted","Data":"7bc5a6a456ec6c62c5539b5002f638e137d2dafb5791cbcc74b7be1d335ab46c"}
Nov 02 13:43:05.837824 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.837819    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" event={"ID":"f46f80e0-332a-4c19-81a2-08cbb65f7fb8","Type":"ContainerStarted","Data":"3de50433774a20cddfa41ddda2e56a91bac00b9a532c092039efeecfc8e9563a"}
Nov 02 13:43:05.837935 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.837868    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" event={"ID":"f46f80e0-332a-4c19-81a2-08cbb65f7fb8","Type":"ContainerStarted","Data":"f7918b0e76d8072d6655bc5cc566db3e5fddf602b0eba23e20f713d381aacb3c"}
Nov 02 13:43:05.837935 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.837916    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" event={"ID":"f46f80e0-332a-4c19-81a2-08cbb65f7fb8","Type":"ContainerStarted","Data":"9b098319bc8df2316cbeef1679a451869a3f25e9ede434fae05c03bf23a369ec"}
Nov 02 13:43:05.894212 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:05.894168    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vnt96" podStartSLOduration=6.41282709 podStartE2EDuration="16.894157288s" totalImagesPullingTime="10.481330198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:05.894040682 +0000 UTC m=+18.687222530" watchObservedRunningTime="2026-11-02 13:43:05.894157288 +0000 UTC m=+18.687339115"
Nov 02 13:43:06.633571 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:06.633543    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:06.633720 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:06.633628    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:06.842189 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:06.842161    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" event={"ID":"f46f80e0-332a-4c19-81a2-08cbb65f7fb8","Type":"ContainerStarted","Data":"51635303ccccdc61ebcf20ea64b259e353f8b68c1ec7f2b61dad088662320225"}
Nov 02 13:43:07.634671 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:07.634640    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:07.634858 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:07.634802    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:07.634858 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:07.634853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:07.634956 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:07.634941    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:08.634419 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:08.634391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:08.634801 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:08.634479    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:08.848396 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:08.848365    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" event={"ID":"f46f80e0-332a-4c19-81a2-08cbb65f7fb8","Type":"ContainerStarted","Data":"a49d63e1a217dbb8d400928f63c37e60f243326469ddebf888d65ea5b4034c9c"}
Nov 02 13:43:09.634049 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:09.634021    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:09.634193 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:09.634060    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:09.634193 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:09.634158    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:09.634318 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:09.634268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:10.634146 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:10.634118    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:10.634804 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:10.634207    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:10.855522 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:10.855493    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" event={"ID":"f46f80e0-332a-4c19-81a2-08cbb65f7fb8","Type":"ContainerStarted","Data":"70d7041237491fa5c1934ae92de506ed00dddd3cef72b3676b27f5e1056d9760"}
Nov 02 13:43:10.855721 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:10.855705    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:10.870010 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:10.869992    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:10.957989 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:10.957917    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k" podStartSLOduration=7.340745769 podStartE2EDuration="22.9579066s" totalImagesPullingTime="15.617160831s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:10.957433372 +0000 UTC m=+23.750615200" watchObservedRunningTime="2026-11-02 13:43:10.9579066 +0000 UTC m=+23.751088427"
Nov 02 13:43:11.197645 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:11.197613    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:11.633771 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:11.633738    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:11.633994 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:11.633869    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:11.633994 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:11.633925    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:11.634060 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:11.634004    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:11.857706 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:11.857681    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:11.872283 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:11.872161    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:12.102410 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:12.102383    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2gjsj"]
Nov 02 13:43:12.102578 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:12.102476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:12.102578 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:12.102555    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:12.105991 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:12.105969    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zp7ql"]
Nov 02 13:43:12.106074 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:12.106060    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:12.106153 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:12.106137    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:12.114342 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:12.114318    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g8jrw"]
Nov 02 13:43:12.114441 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:12.114413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:12.114513 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:12.114494    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:13.633851 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:13.633829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:13.634265 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:13.633829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:13.634265 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:13.633927    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:13.634265 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:13.633996    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:13.634265 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:13.634033    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:13.634265 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:13.634090    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:14.193197 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:14.193172    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:43:14.193360 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:14.193308    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:14.193746 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:14.193715    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:43:15.634043 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:15.634008    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:15.634485 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:15.634122    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:15.634485 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:15.634145    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:15.634485 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:15.634188    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:15.634485 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:15.634248    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:15.634485 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:15.634311    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:17.635069 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:17.635042    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:17.635440 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:17.635142    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8jrw" podUID="91600010-0a54-4517-b9a7-1cc38b075abd"
Nov 02 13:43:17.635440 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:17.635216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:17.635440 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:17.635296    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2gjsj" podUID="fb318d8f-e7de-4185-96cd-d592b304248a"
Nov 02 13:43:17.635440 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:17.635334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:17.635440 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:17.635402    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zp7ql" podUID="00be7ba2-6f06-4ed5-a007-d0ac612177ee"
Nov 02 13:43:17.968413 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:17.968354    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeReady"
Nov 02 13:43:17.968523 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:17.968442    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 13:43:18.023099 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.023073    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-75dw4"]
Nov 02 13:43:18.061877 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.061856    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"]
Nov 02 13:43:18.061990 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.061973    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-75dw4"
Nov 02 13:43:18.075581 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.075558    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zlk72"]
Nov 02 13:43:18.075680 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.075664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.085307 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.085107    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.085307 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.085111    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.087346 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.087327    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pm5xb"]
Nov 02 13:43:18.087469 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.087402    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zlk72"
Nov 02 13:43:18.094404 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.094385    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 13:43:18.096353 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.095485    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"]
Nov 02 13:43:18.096353 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.096191    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pm5xb"
Nov 02 13:43:18.121834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.121812    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fzlvp"]
Nov 02 13:43:18.121939 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.121877    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.137606 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.137586    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-f79wg"]
Nov 02 13:43:18.137705 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.137696    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.171403 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.171385    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5466575dd5-gsvt5"]
Nov 02 13:43:18.171500 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.171486    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.184383 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.184364    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.184517 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.184490    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.184517 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.184503    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.184517 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.184512    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"]
Nov 02 13:43:18.184712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.184523    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.184712 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.184639    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.184818 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.184747    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 13:43:18.184818 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.184749    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 13:43:18.184947 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.184931    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 13:43:18.192379 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.192364    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d555fdbf5-mnstt"]
Nov 02 13:43:18.192448 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.192433    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"
Nov 02 13:43:18.199982 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.199961    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.200059 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.199997    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.200059 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.200007    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 13:43:18.200312 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.200288    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 13:43:18.200312 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.200308    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 13:43:18.200441 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.200314    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 13:43:18.200441 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.200370    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hkz22\""
Nov 02 13:43:18.201025 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.201002    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 13:43:18.201726 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.201713    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.201726 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.201724    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.201822 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.201794    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 13:43:18.201984 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.201971    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 13:43:18.202317 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.202303    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 13:43:18.202354 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.202320    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 13:43:18.202493 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.202480    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 13:43:18.205757 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.205741    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"]
Nov 02 13:43:18.205884 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.205861    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.213846 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.213826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.213944 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.213866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.213944 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.213891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4p82\" (UniqueName: \"kubernetes.io/projected/63881d4e-a3bf-4948-a369-c892baba23a8-kube-api-access-f4p82\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.214019 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.213970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxts2\" (UniqueName: \"kubernetes.io/projected/398a3f5c-7cbe-448d-8f16-3360c18a38bf-kube-api-access-xxts2\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.214063 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-trusted-ca\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.214114 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrlq\" (UniqueName: \"kubernetes.io/projected/18d90fc8-23b1-4fca-a9e2-83eb8f4786d4-kube-api-access-qdrlq\") pod \"kas-connection-checker-685895c7c9-pm5xb\" (UID: \"18d90fc8-23b1-4fca-a9e2-83eb8f4786d4\") " pod="kube-system/kas-connection-checker-685895c7c9-pm5xb"
Nov 02 13:43:18.214200 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214183    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/398a3f5c-7cbe-448d-8f16-3360c18a38bf-serving-cert\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.214241 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-config\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.214282 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.214282 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214266    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhlq\" (UniqueName: \"kubernetes.io/projected/35f03797-afb7-4aa5-a841-c665e99feaa4-kube-api-access-tzhlq\") pod \"kas-connection-checker-685895c7c9-zlk72\" (UID: \"35f03797-afb7-4aa5-a841-c665e99feaa4\") " pod="kube-system/kas-connection-checker-685895c7c9-zlk72"
Nov 02 13:43:18.214407 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214391    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63881d4e-a3bf-4948-a369-c892baba23a8-config-volume\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.214455 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcbk\" (UniqueName: \"kubernetes.io/projected/3bfd00d2-35ee-4067-8649-bdb54ee596a2-kube-api-access-fzcbk\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.214486 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/8816a23c-15b8-4775-8853-6dc6cf7526c5-kube-api-access-qjlx2\") pod \"kas-connection-checker-685895c7c9-75dw4\" (UID: \"8816a23c-15b8-4775-8853-6dc6cf7526c5\") " pod="kube-system/kas-connection-checker-685895c7c9-75dw4"
Nov 02 13:43:18.214520 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.214483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63881d4e-a3bf-4948-a369-c892baba23a8-tmp-dir\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.221572 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.221536    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8c6787bcd-psbdk"]
Nov 02 13:43:18.221645 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.221632    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:18.256179 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.256164    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"]
Nov 02 13:43:18.256292 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.256276    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.266508 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.266492    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"]
Nov 02 13:43:18.266621 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.266605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:18.268828 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.268812    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"]
Nov 02 13:43:18.268949 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.268930    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.274803 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.274771    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b7nn8"]
Nov 02 13:43:18.274906 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.274852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.279180 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.279162    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"]
Nov 02 13:43:18.279311 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.279296    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:18.284491 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.284471    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 13:43:18.284570 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.284489    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.284570 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.284501    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.284802 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.284768    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.284865 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.284822    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.285581 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.285564    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.285685 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.285597    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 13:43:18.285685 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.285616    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.285685 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.285678    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 13:43:18.285828 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.285756    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 13:43:18.291973 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.291954    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 13:43:18.292590 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292573    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-75dw4"]
Nov 02 13:43:18.292667 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292608    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zlk72"]
Nov 02 13:43:18.292667 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292630    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"]
Nov 02 13:43:18.292667 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292653    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pm5xb"]
Nov 02 13:43:18.292834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292671    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"]
Nov 02 13:43:18.292834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292687    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"
Nov 02 13:43:18.292834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292696    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"]
Nov 02 13:43:18.292834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292717    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"]
Nov 02 13:43:18.292834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292736    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-f79wg"]
Nov 02 13:43:18.292834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292757    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"]
Nov 02 13:43:18.292834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292797    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7nn8"]
Nov 02 13:43:18.292834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292819    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"]
Nov 02 13:43:18.293171 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292839    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"]
Nov 02 13:43:18.293171 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292859    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"]
Nov 02 13:43:18.293171 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292878    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5466575dd5-gsvt5"]
Nov 02 13:43:18.293171 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292897    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fzlvp"]
Nov 02 13:43:18.293171 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292928    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d555fdbf5-mnstt"]
Nov 02 13:43:18.293171 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.292950    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8c6787bcd-psbdk"]
Nov 02 13:43:18.294654 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.294635    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 13:43:18.294740 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.294666    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 13:43:18.294740 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.294667    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 13:43:18.294740 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.294688    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 13:43:18.294740 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.294718    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 13:43:18.294740 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.294722    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 13:43:18.294957 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.294842    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 13:43:18.294957 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.294865    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sk6c2\""
Nov 02 13:43:18.294957 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.294889    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cjxg7\""
Nov 02 13:43:18.296070 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.296050    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wff6\""
Nov 02 13:43:18.296070 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.296070    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 13:43:18.315437 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpj5\" (UniqueName: \"kubernetes.io/projected/8646ac2d-cfd2-4b7c-929a-906815e6551f-kube-api-access-dkpj5\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.315524 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315445    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-default-certificate\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.315524 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.315524 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4p82\" (UniqueName: \"kubernetes.io/projected/63881d4e-a3bf-4948-a369-c892baba23a8-kube-api-access-f4p82\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.315635 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.315552    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:43:18.315635 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.315599    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls podName:63881d4e-a3bf-4948-a369-c892baba23a8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:18.815587025 +0000 UTC m=+31.608768874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls") pod "dns-default-fzlvp" (UID: "63881d4e-a3bf-4948-a369-c892baba23a8") : secret "dns-default-metrics-tls" not found
Nov 02 13:43:18.315635 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.315635 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315626    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrlq\" (UniqueName: \"kubernetes.io/projected/18d90fc8-23b1-4fca-a9e2-83eb8f4786d4-kube-api-access-qdrlq\") pod \"kas-connection-checker-685895c7c9-pm5xb\" (UID: \"18d90fc8-23b1-4fca-a9e2-83eb8f4786d4\") " pod="kube-system/kas-connection-checker-685895c7c9-pm5xb"
Nov 02 13:43:18.315841 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315648    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.315841 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.315841 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.315841 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhlq\" (UniqueName: \"kubernetes.io/projected/35f03797-afb7-4aa5-a841-c665e99feaa4-kube-api-access-tzhlq\") pod \"kas-connection-checker-685895c7c9-zlk72\" (UID: \"35f03797-afb7-4aa5-a841-c665e99feaa4\") " pod="kube-system/kas-connection-checker-685895c7c9-zlk72"
Nov 02 13:43:18.315841 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.316031 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315857    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-config\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.316031 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.315868    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 13:43:18.316031 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b34b2c6f-210f-4256-8383-16f6a3a360bb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:18.316031 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.315923    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls podName:3bfd00d2-35ee-4067-8649-bdb54ee596a2 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:18.815910048 +0000 UTC m=+31.609091880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5ssg4" (UID: "3bfd00d2-35ee-4067-8649-bdb54ee596a2") : secret "samples-operator-tls" not found
Nov 02 13:43:18.316031 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-tmp\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.316031 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.315976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.316031 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63881d4e-a3bf-4948-a369-c892baba23a8-config-volume\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.316263 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316045    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63881d4e-a3bf-4948-a369-c892baba23a8-tmp-dir\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.316263 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.316263 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316095    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-snapshots\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.316263 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316112    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xskc\" (UniqueName: \"kubernetes.io/projected/d861fac3-737a-4cd7-abc6-4af75b4c1d67-kube-api-access-5xskc\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.316263 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.316263 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316166    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4fv4\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.316263 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316186    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:18.316511 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxts2\" (UniqueName: \"kubernetes.io/projected/398a3f5c-7cbe-448d-8f16-3360c18a38bf-kube-api-access-xxts2\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.316511 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316324    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-stats-auth\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.316511 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-trusted-ca\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.316511 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316378    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63881d4e-a3bf-4948-a369-c892baba23a8-tmp-dir\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.316511 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk7vv\" (UniqueName: \"kubernetes.io/projected/4150923a-78f1-4f52-a0d5-271ec942bd1b-kube-api-access-jk7vv\") pod \"volume-data-source-validator-58c9f5f46d-np42z\" (UID: \"4150923a-78f1-4f52-a0d5-271ec942bd1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"
Nov 02 13:43:18.316511 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.316511 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316479    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/398a3f5c-7cbe-448d-8f16-3360c18a38bf-serving-cert\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.316511 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316507    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-config\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.316904 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-service-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.316904 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.316904 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.316904 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316597    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63881d4e-a3bf-4948-a369-c892baba23a8-config-volume\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.316904 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.316904 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316662    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8646ac2d-cfd2-4b7c-929a-906815e6551f-serving-cert\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.316904 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcbk\" (UniqueName: \"kubernetes.io/projected/3bfd00d2-35ee-4067-8649-bdb54ee596a2-kube-api-access-fzcbk\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.316904 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/8816a23c-15b8-4775-8853-6dc6cf7526c5-kube-api-access-qjlx2\") pod \"kas-connection-checker-685895c7c9-75dw4\" (UID: \"8816a23c-15b8-4775-8853-6dc6cf7526c5\") " pod="kube-system/kas-connection-checker-685895c7c9-75dw4"
Nov 02 13:43:18.316904 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.316877    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.317321 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.317246    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-trusted-ca\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.319756 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.319739    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/398a3f5c-7cbe-448d-8f16-3360c18a38bf-serving-cert\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.344080 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.344054    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4p82\" (UniqueName: \"kubernetes.io/projected/63881d4e-a3bf-4948-a369-c892baba23a8-kube-api-access-f4p82\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.344183 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.344101    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxts2\" (UniqueName: \"kubernetes.io/projected/398a3f5c-7cbe-448d-8f16-3360c18a38bf-kube-api-access-xxts2\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.344263 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.344242    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhlq\" (UniqueName: \"kubernetes.io/projected/35f03797-afb7-4aa5-a841-c665e99feaa4-kube-api-access-tzhlq\") pod \"kas-connection-checker-685895c7c9-zlk72\" (UID: \"35f03797-afb7-4aa5-a841-c665e99feaa4\") " pod="kube-system/kas-connection-checker-685895c7c9-zlk72"
Nov 02 13:43:18.344605 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.344583    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/8816a23c-15b8-4775-8853-6dc6cf7526c5-kube-api-access-qjlx2\") pod \"kas-connection-checker-685895c7c9-75dw4\" (UID: \"8816a23c-15b8-4775-8853-6dc6cf7526c5\") " pod="kube-system/kas-connection-checker-685895c7c9-75dw4"
Nov 02 13:43:18.344843 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.344827    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcbk\" (UniqueName: \"kubernetes.io/projected/3bfd00d2-35ee-4067-8649-bdb54ee596a2-kube-api-access-fzcbk\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.345012 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.344996    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrlq\" (UniqueName: \"kubernetes.io/projected/18d90fc8-23b1-4fca-a9e2-83eb8f4786d4-kube-api-access-qdrlq\") pod \"kas-connection-checker-685895c7c9-pm5xb\" (UID: \"18d90fc8-23b1-4fca-a9e2-83eb8f4786d4\") " pod="kube-system/kas-connection-checker-685895c7c9-pm5xb"
Nov 02 13:43:18.384892 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.384867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-75dw4"
Nov 02 13:43:18.384963 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.384892    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.385067 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.385044    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.385175 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.385097    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.385175 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.385109    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 13:43:18.385175 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.385148    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.385175 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.385168    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:43:18.393750 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.393733    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:18.417023 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.416999    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.417108 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417038    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb81289b-9a76-4529-a6f5-649ea7c85d67-serving-cert\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.417108 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417072    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-snapshots\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.417219 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xskc\" (UniqueName: \"kubernetes.io/projected/d861fac3-737a-4cd7-abc6-4af75b4c1d67-kube-api-access-5xskc\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.417219 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.417219 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4fv4\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.417354 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:18.417354 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.417354 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417290    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.417354 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-stats-auth\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.417536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrcq\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.417536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.417536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.417536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk7vv\" (UniqueName: \"kubernetes.io/projected/4150923a-78f1-4f52-a0d5-271ec942bd1b-kube-api-access-jk7vv\") pod \"volume-data-source-validator-58c9f5f46d-np42z\" (UID: \"4150923a-78f1-4f52-a0d5-271ec942bd1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"
Nov 02 13:43:18.417536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.417536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27e3e0a4-dc29-4e68-9344-f82ea07cb067-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.417536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417528    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zld\" (UniqueName: \"kubernetes.io/projected/df16fea1-6bca-47f3-b9bd-35716e3d50f3-kube-api-access-v9zld\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:18.417891 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417559    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-service-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.417891 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417582    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-snapshots\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.417891 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.418037 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.418037 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.418037 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.417974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8646ac2d-cfd2-4b7c-929a-906815e6551f-serving-cert\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.418037 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.418037 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpj5\" (UniqueName: \"kubernetes.io/projected/8646ac2d-cfd2-4b7c-929a-906815e6551f-kube-api-access-dkpj5\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.418266 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb81289b-9a76-4529-a6f5-649ea7c85d67-config\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.418266 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418086    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-default-certificate\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.418266 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhvr\" (UniqueName: \"kubernetes.io/projected/fb81289b-9a76-4529-a6f5-649ea7c85d67-kube-api-access-vlhvr\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.418266 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418155    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.418266 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58czq\" (UniqueName: \"kubernetes.io/projected/27e3e0a4-dc29-4e68-9344-f82ea07cb067-kube-api-access-58czq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.418266 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.418536 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.418326    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 13:43:18.418536 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.418373    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert podName:b34b2c6f-210f-4256-8383-16f6a3a360bb nodeName:}" failed. No retries permitted until 2026-11-02 13:43:18.918358434 +0000 UTC m=+31.711540254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jc4gv" (UID: "b34b2c6f-210f-4256-8383-16f6a3a360bb") : secret "networking-console-plugin-cert" not found
Nov 02 13:43:18.418536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.418536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:18.418536 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418447    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.418803 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.418681    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:18.918667728 +0000 UTC m=+31.711849550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : configmap references non-existent config key: service-ca.crt
Nov 02 13:43:18.418803 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.419003 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.418974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.419355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.419330    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.419630 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.419614    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:18.419630 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.419632    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d555fdbf5-mnstt: secret "image-registry-tls" not found
Nov 02 13:43:18.419830 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.419673    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls podName:3fa80f89-f24d-4f86-940c-864dbf177a5c nodeName:}" failed. No retries permitted until 2026-11-02 13:43:18.919659475 +0000 UTC m=+31.712841324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls") pod "image-registry-5d555fdbf5-mnstt" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c") : secret "image-registry-tls" not found
Nov 02 13:43:18.419830 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.419756    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.420009 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.419966    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.420094 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.420077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b34b2c6f-210f-4256-8383-16f6a3a360bb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:18.420646 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.420605    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-service-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.420747 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.420642    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2ca56ffb-997f-4edb-9efa-0609042e8f08-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:18.421004 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.420980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-tmp\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.421089 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.421047    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.421207 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.421076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vw64\" (UniqueName: \"kubernetes.io/projected/2ca56ffb-997f-4edb-9efa-0609042e8f08-kube-api-access-8vw64\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:18.421207 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.421131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27e3e0a4-dc29-4e68-9344-f82ea07cb067-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.421207 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.421153    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df16fea1-6bca-47f3-b9bd-35716e3d50f3-cert\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:18.421348 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.421214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs7rc\" (UniqueName: \"kubernetes.io/projected/488bddc1-06e8-446c-b612-5ddeac1d8bdf-kube-api-access-rs7rc\") pod \"network-check-source-6f78d59f99-nkjt5\" (UID: \"488bddc1-06e8-446c-b612-5ddeac1d8bdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"
Nov 02 13:43:18.421348 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.421286    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b34b2c6f-210f-4256-8383-16f6a3a360bb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:18.421348 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.421315    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 13:43:18.421492 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.421357    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:18.921344193 +0000 UTC m=+31.714526020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : secret "router-metrics-certs-default" not found
Nov 02 13:43:18.421591 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.421571    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-tmp\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.421942 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.421925    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.422069 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.422002    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zlk72"
Nov 02 13:43:18.422163 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.422100    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-default-certificate\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.423117 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.423090    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8646ac2d-cfd2-4b7c-929a-906815e6551f-serving-cert\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.423207 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.423177    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.423443 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.423426    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-stats-auth\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.437411 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.437384    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pm5xb"
Nov 02 13:43:18.445022 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.445001    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.445314 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.445294    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk7vv\" (UniqueName: \"kubernetes.io/projected/4150923a-78f1-4f52-a0d5-271ec942bd1b-kube-api-access-jk7vv\") pod \"volume-data-source-validator-58c9f5f46d-np42z\" (UID: \"4150923a-78f1-4f52-a0d5-271ec942bd1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"
Nov 02 13:43:18.445502 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.445486    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xskc\" (UniqueName: \"kubernetes.io/projected/d861fac3-737a-4cd7-abc6-4af75b4c1d67-kube-api-access-5xskc\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.446030 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.446012    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4fv4\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.448166 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.448146    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpj5\" (UniqueName: \"kubernetes.io/projected/8646ac2d-cfd2-4b7c-929a-906815e6551f-kube-api-access-dkpj5\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.492370 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.491455    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:18.522369 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.522338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2ca56ffb-997f-4edb-9efa-0609042e8f08-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:18.522460 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.522422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vw64\" (UniqueName: \"kubernetes.io/projected/2ca56ffb-997f-4edb-9efa-0609042e8f08-kube-api-access-8vw64\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27e3e0a4-dc29-4e68-9344-f82ea07cb067-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df16fea1-6bca-47f3-b9bd-35716e3d50f3-cert\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs7rc\" (UniqueName: \"kubernetes.io/projected/488bddc1-06e8-446c-b612-5ddeac1d8bdf-kube-api-access-rs7rc\") pod \"network-check-source-6f78d59f99-nkjt5\" (UID: \"488bddc1-06e8-446c-b612-5ddeac1d8bdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb81289b-9a76-4529-a6f5-649ea7c85d67-serving-cert\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrcq\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27e3e0a4-dc29-4e68-9344-f82ea07cb067-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zld\" (UniqueName: \"kubernetes.io/projected/df16fea1-6bca-47f3-b9bd-35716e3d50f3-kube-api-access-v9zld\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523442    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2ca56ffb-997f-4edb-9efa-0609042e8f08-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb81289b-9a76-4529-a6f5-649ea7c85d67-config\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.525355 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhvr\" (UniqueName: \"kubernetes.io/projected/fb81289b-9a76-4529-a6f5-649ea7c85d67-kube-api-access-vlhvr\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523585    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58czq\" (UniqueName: \"kubernetes.io/projected/27e3e0a4-dc29-4e68-9344-f82ea07cb067-kube-api-access-58czq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523659    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27e3e0a4-dc29-4e68-9344-f82ea07cb067-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.523921    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.523986    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.524055    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls podName:2ca56ffb-997f-4edb-9efa-0609042e8f08 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.024037542 +0000 UTC m=+31.817219350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nnxtg" (UID: "2ca56ffb-997f-4edb-9efa-0609042e8f08") : secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.524078    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.524522    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.524538    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8c6787bcd-psbdk: secret "image-registry-tls" not found
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.524582    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls podName:633c4858-9dc5-4049-aec3-8a14171652ae nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.024566616 +0000 UTC m=+31.817748425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls") pod "image-registry-8c6787bcd-psbdk" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae") : secret "image-registry-tls" not found
Nov 02 13:43:18.526251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.525244    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb81289b-9a76-4529-a6f5-649ea7c85d67-config\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.528289 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.527345    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.529718 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.529604    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27e3e0a4-dc29-4e68-9344-f82ea07cb067-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.530091 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.529915    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.530852 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.530385    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb81289b-9a76-4529-a6f5-649ea7c85d67-serving-cert\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.532154 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.532131    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.532573 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.532516    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df16fea1-6bca-47f3-b9bd-35716e3d50f3-cert\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:18.546348 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.546180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"
Nov 02 13:43:18.551073 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.551020    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vw64\" (UniqueName: \"kubernetes.io/projected/2ca56ffb-997f-4edb-9efa-0609042e8f08-kube-api-access-8vw64\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:18.552164 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.552138    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs7rc\" (UniqueName: \"kubernetes.io/projected/488bddc1-06e8-446c-b612-5ddeac1d8bdf-kube-api-access-rs7rc\") pod \"network-check-source-6f78d59f99-nkjt5\" (UID: \"488bddc1-06e8-446c-b612-5ddeac1d8bdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"
Nov 02 13:43:18.553612 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.553591    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zld\" (UniqueName: \"kubernetes.io/projected/df16fea1-6bca-47f3-b9bd-35716e3d50f3-kube-api-access-v9zld\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:18.553892 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.553869    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrcq\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.554753 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.554703    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58czq\" (UniqueName: \"kubernetes.io/projected/27e3e0a4-dc29-4e68-9344-f82ea07cb067-kube-api-access-58czq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.554753 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.554724    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:18.555078 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.555059    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhvr\" (UniqueName: \"kubernetes.io/projected/fb81289b-9a76-4529-a6f5-649ea7c85d67-kube-api-access-vlhvr\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.603213 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.603188    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:18.627128 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.627106    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:18.630261 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.630243    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:18.633409 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.633371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"
Nov 02 13:43:18.722321 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.722244    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-75dw4"]
Nov 02 13:43:18.726079 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.724415    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"]
Nov 02 13:43:18.769039 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.769009    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zlk72"]
Nov 02 13:43:18.789143 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.789116    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pm5xb"]
Nov 02 13:43:18.828883 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.828078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:18.828883 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.828161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:18.828883 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.828425    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 13:43:18.828883 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.828486    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls podName:3bfd00d2-35ee-4067-8649-bdb54ee596a2 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.828466938 +0000 UTC m=+32.621648765 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5ssg4" (UID: "3bfd00d2-35ee-4067-8649-bdb54ee596a2") : secret "samples-operator-tls" not found
Nov 02 13:43:18.828883 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.828863    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:43:18.829163 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.828911    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls podName:63881d4e-a3bf-4948-a369-c892baba23a8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.828896281 +0000 UTC m=+32.622078107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls") pod "dns-default-fzlvp" (UID: "63881d4e-a3bf-4948-a369-c892baba23a8") : secret "dns-default-metrics-tls" not found
Nov 02 13:43:18.830236 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.830151    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-f79wg"]
Nov 02 13:43:18.831025 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:18.830998    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8646ac2d_cfd2_4b7c_929a_906815e6551f.slice/crio-f053214b2a15076cb42798094959229a8ce7676b8149a25e06089dd398798da5 WatchSource:0}: Error finding container f053214b2a15076cb42798094959229a8ce7676b8149a25e06089dd398798da5: Status 404 returned error can't find the container with id f053214b2a15076cb42798094959229a8ce7676b8149a25e06089dd398798da5
Nov 02 13:43:18.873894 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.873865    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-f79wg" event={"ID":"8646ac2d-cfd2-4b7c-929a-906815e6551f","Type":"ContainerStarted","Data":"f053214b2a15076cb42798094959229a8ce7676b8149a25e06089dd398798da5"}
Nov 02 13:43:18.875514 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.875422    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"]
Nov 02 13:43:18.877543 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.876740    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pm5xb" event={"ID":"18d90fc8-23b1-4fca-a9e2-83eb8f4786d4","Type":"ContainerStarted","Data":"de8e7e42dd03bdf1530b5ac4d7bb83c122e4ff195ea9d67b197e983390925b42"}
Nov 02 13:43:18.877950 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:18.877764    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4150923a_78f1_4f52_a0d5_271ec942bd1b.slice/crio-69e32cae509e5d6868d134c1f4a1a1ba296d495157dbf4690787a29120c17b79 WatchSource:0}: Error finding container 69e32cae509e5d6868d134c1f4a1a1ba296d495157dbf4690787a29120c17b79: Status 404 returned error can't find the container with id 69e32cae509e5d6868d134c1f4a1a1ba296d495157dbf4690787a29120c17b79
Nov 02 13:43:18.883138 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.882658    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zlk72" event={"ID":"35f03797-afb7-4aa5-a841-c665e99feaa4","Type":"ContainerStarted","Data":"1c95932f9108f4aed9cb7db3ab6750457f8867c4414da68250c7132e4419b6a5"}
Nov 02 13:43:18.885021 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.884993    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" event={"ID":"398a3f5c-7cbe-448d-8f16-3360c18a38bf","Type":"ContainerStarted","Data":"b04e07673f0d267b355ade260ac8f3ff163299191886025e39e949e1d00a3e05"}
Nov 02 13:43:18.886481 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.886452    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-75dw4" event={"ID":"8816a23c-15b8-4775-8853-6dc6cf7526c5","Type":"ContainerStarted","Data":"6db9f5bd2d36b7fdca6a5b46f0deb14825a11f1e91be741e5d8db248f90a8706"}
Nov 02 13:43:18.929460 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.929436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.929549 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.929500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:18.929549 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.929530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:18.929649 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.929572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:18.929699 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.929683    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:18.929699 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.929692    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d555fdbf5-mnstt: secret "image-registry-tls" not found
Nov 02 13:43:18.929817 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.929725    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls podName:3fa80f89-f24d-4f86-940c-864dbf177a5c nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.929714115 +0000 UTC m=+32.722895934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls") pod "image-registry-5d555fdbf5-mnstt" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c") : secret "image-registry-tls" not found
Nov 02 13:43:18.929817 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.929773    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.929766561 +0000 UTC m=+32.722948380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : configmap references non-existent config key: service-ca.crt
Nov 02 13:43:18.929926 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.929842    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 13:43:18.929926 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.929865    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.92985868 +0000 UTC m=+32.723040498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : secret "router-metrics-certs-default" not found
Nov 02 13:43:18.929926 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.929902    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 13:43:18.929926 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:18.929924    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert podName:b34b2c6f-210f-4256-8383-16f6a3a360bb nodeName:}" failed. No retries permitted until 2026-11-02 13:43:19.929918656 +0000 UTC m=+32.723100474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jc4gv" (UID: "b34b2c6f-210f-4256-8383-16f6a3a360bb") : secret "networking-console-plugin-cert" not found
Nov 02 13:43:18.941133 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:18.941103    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27e3e0a4_dc29_4e68_9344_f82ea07cb067.slice/crio-3fd128a0f6e080fd4646150fee33128327829dd6ae32ca4797f76c8e35287cc9 WatchSource:0}: Error finding container 3fd128a0f6e080fd4646150fee33128327829dd6ae32ca4797f76c8e35287cc9: Status 404 returned error can't find the container with id 3fd128a0f6e080fd4646150fee33128327829dd6ae32ca4797f76c8e35287cc9
Nov 02 13:43:18.943159 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.943142    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"]
Nov 02 13:43:18.975203 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:18.975181    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb81289b_9a76_4529_a6f5_649ea7c85d67.slice/crio-8cbfe03b6a690c4bc8c1215b4b789a1c6ab5bbbc9ddbae9b197d765c8b3cab8c WatchSource:0}: Error finding container 8cbfe03b6a690c4bc8c1215b4b789a1c6ab5bbbc9ddbae9b197d765c8b3cab8c: Status 404 returned error can't find the container with id 8cbfe03b6a690c4bc8c1215b4b789a1c6ab5bbbc9ddbae9b197d765c8b3cab8c
Nov 02 13:43:18.975600 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:18.975581    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"]
Nov 02 13:43:19.000752 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.000730    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"]
Nov 02 13:43:19.001021 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:19.000998    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488bddc1_06e8_446c_b612_5ddeac1d8bdf.slice/crio-93d9de324f0fadd867b9df3293b8f56a504c91e4f551b8db56b824d5da69fc65 WatchSource:0}: Error finding container 93d9de324f0fadd867b9df3293b8f56a504c91e4f551b8db56b824d5da69fc65: Status 404 returned error can't find the container with id 93d9de324f0fadd867b9df3293b8f56a504c91e4f551b8db56b824d5da69fc65
Nov 02 13:43:19.028888 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.028866    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7nn8"]
Nov 02 13:43:19.029301 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:19.029281    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf16fea1_6bca_47f3_b9bd_35716e3d50f3.slice/crio-1a181f38f91d1d321f8247113e0d9fd13a042aa0786d7a0eccaed1c82feb4130 WatchSource:0}: Error finding container 1a181f38f91d1d321f8247113e0d9fd13a042aa0786d7a0eccaed1c82feb4130: Status 404 returned error can't find the container with id 1a181f38f91d1d321f8247113e0d9fd13a042aa0786d7a0eccaed1c82feb4130
Nov 02 13:43:19.030307 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.030283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:19.030420 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.030406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:19.030460 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.030429    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:19.030498 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.030479    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls podName:2ca56ffb-997f-4edb-9efa-0609042e8f08 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:20.030465434 +0000 UTC m=+32.823647244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nnxtg" (UID: "2ca56ffb-997f-4edb-9efa-0609042e8f08") : secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:19.030548 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.030540    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:19.030589 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.030553    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8c6787bcd-psbdk: secret "image-registry-tls" not found
Nov 02 13:43:19.030669 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.030601    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls podName:633c4858-9dc5-4049-aec3-8a14171652ae nodeName:}" failed. No retries permitted until 2026-11-02 13:43:20.030585968 +0000 UTC m=+32.823767791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls") pod "image-registry-8c6787bcd-psbdk" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae") : secret "image-registry-tls" not found
Nov 02 13:43:19.635750 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.635628    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:19.637062 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.637036    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:19.637264 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.637242    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:19.654351 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.654322    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fs4v4\""
Nov 02 13:43:19.654454 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.654408    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xwmgj\""
Nov 02 13:43:19.654454 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.654448    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lj29d\""
Nov 02 13:43:19.654561 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.654497    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:43:19.654561 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.654522    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:43:19.839398 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.838836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:19.839398 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.838922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:19.839398 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.839001    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:43:19.839398 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.839057    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 13:43:19.839398 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.839072    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls podName:63881d4e-a3bf-4948-a369-c892baba23a8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:21.839052805 +0000 UTC m=+34.632234645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls") pod "dns-default-fzlvp" (UID: "63881d4e-a3bf-4948-a369-c892baba23a8") : secret "dns-default-metrics-tls" not found
Nov 02 13:43:19.839398 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.839096    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls podName:3bfd00d2-35ee-4067-8649-bdb54ee596a2 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:21.839084155 +0000 UTC m=+34.632265984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5ssg4" (UID: "3bfd00d2-35ee-4067-8649-bdb54ee596a2") : secret "samples-operator-tls" not found
Nov 02 13:43:19.893834 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.893695    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7nn8" event={"ID":"df16fea1-6bca-47f3-b9bd-35716e3d50f3","Type":"ContainerStarted","Data":"1a181f38f91d1d321f8247113e0d9fd13a042aa0786d7a0eccaed1c82feb4130"}
Nov 02 13:43:19.896392 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.896321    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n" event={"ID":"fb81289b-9a76-4529-a6f5-649ea7c85d67","Type":"ContainerStarted","Data":"8cbfe03b6a690c4bc8c1215b4b789a1c6ab5bbbc9ddbae9b197d765c8b3cab8c"}
Nov 02 13:43:19.907971 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.907938    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pm5xb" event={"ID":"18d90fc8-23b1-4fca-a9e2-83eb8f4786d4","Type":"ContainerStarted","Data":"5c58d155110009ba7afe75769248b2b62bad30d0e6b3b904d0b360dfe6e48837"}
Nov 02 13:43:19.922018 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.921991    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zlk72" event={"ID":"35f03797-afb7-4aa5-a841-c665e99feaa4","Type":"ContainerStarted","Data":"207742e1f13df6a153045af04629af73ee379103b4f5739eb91daf5eacfca679"}
Nov 02 13:43:19.926285 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.926259    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5" event={"ID":"488bddc1-06e8-446c-b612-5ddeac1d8bdf","Type":"ContainerStarted","Data":"93d9de324f0fadd867b9df3293b8f56a504c91e4f551b8db56b824d5da69fc65"}
Nov 02 13:43:19.928495 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.928472    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z" event={"ID":"4150923a-78f1-4f52-a0d5-271ec942bd1b","Type":"ContainerStarted","Data":"69e32cae509e5d6868d134c1f4a1a1ba296d495157dbf4690787a29120c17b79"}
Nov 02 13:43:19.932414 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.932390    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6" event={"ID":"27e3e0a4-dc29-4e68-9344-f82ea07cb067","Type":"ContainerStarted","Data":"3fd128a0f6e080fd4646150fee33128327829dd6ae32ca4797f76c8e35287cc9"}
Nov 02 13:43:19.934189 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.934168    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-75dw4" event={"ID":"8816a23c-15b8-4775-8853-6dc6cf7526c5","Type":"ContainerStarted","Data":"6dd2fbcd5c4c6c5504f844bb846ca20d0bb9b8768f62e01f11f87a0f8ad64fcb"}
Nov 02 13:43:19.939787 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.939751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:19.939867 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.939824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:19.939919 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.939893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:19.939995 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.939980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:19.940153 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.940139    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:21.940106403 +0000 UTC m=+34.733288222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : configmap references non-existent config key: service-ca.crt
Nov 02 13:43:19.940500 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.940472    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 13:43:19.940575 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.940534    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:21.940520563 +0000 UTC m=+34.733702388 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : secret "router-metrics-certs-default" not found
Nov 02 13:43:19.940637 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.940600    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 13:43:19.940637 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.940635    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert podName:b34b2c6f-210f-4256-8383-16f6a3a360bb nodeName:}" failed. No retries permitted until 2026-11-02 13:43:21.94062399 +0000 UTC m=+34.733805814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jc4gv" (UID: "b34b2c6f-210f-4256-8383-16f6a3a360bb") : secret "networking-console-plugin-cert" not found
Nov 02 13:43:19.940748 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.940697    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:19.940748 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.940708    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d555fdbf5-mnstt: secret "image-registry-tls" not found
Nov 02 13:43:19.940748 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:19.940740    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls podName:3fa80f89-f24d-4f86-940c-864dbf177a5c nodeName:}" failed. No retries permitted until 2026-11-02 13:43:21.940729529 +0000 UTC m=+34.733911348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls") pod "image-registry-5d555fdbf5-mnstt" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c") : secret "image-registry-tls" not found
Nov 02 13:43:19.961821 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:19.961751    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pm5xb" podStartSLOduration=386.961734473 podStartE2EDuration="6m26.961734473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:19.961602435 +0000 UTC m=+32.754784264" watchObservedRunningTime="2026-11-02 13:43:19.961734473 +0000 UTC m=+32.754916304"
Nov 02 13:43:20.040773 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:20.040732    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:20.040937 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:20.040887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:20.042637 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:20.041948    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:20.042637 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:20.041970    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8c6787bcd-psbdk: secret "image-registry-tls" not found
Nov 02 13:43:20.042637 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:20.042049    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls podName:633c4858-9dc5-4049-aec3-8a14171652ae nodeName:}" failed. No retries permitted until 2026-11-02 13:43:22.042031334 +0000 UTC m=+34.835213169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls") pod "image-registry-8c6787bcd-psbdk" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae") : secret "image-registry-tls" not found
Nov 02 13:43:20.042637 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:20.042493    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:20.042637 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:20.042525    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls podName:2ca56ffb-997f-4edb-9efa-0609042e8f08 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:22.042515673 +0000 UTC m=+34.835697492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nnxtg" (UID: "2ca56ffb-997f-4edb-9efa-0609042e8f08") : secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:20.075840 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:20.075770    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-75dw4" podStartSLOduration=387.07575383 podStartE2EDuration="6m27.07575383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:20.017334136 +0000 UTC m=+32.810515965" watchObservedRunningTime="2026-11-02 13:43:20.07575383 +0000 UTC m=+32.868935659"
Nov 02 13:43:20.076169 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:20.076138    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zlk72" podStartSLOduration=387.076131468 podStartE2EDuration="6m27.076131468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:36:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:20.075163616 +0000 UTC m=+32.868345445" watchObservedRunningTime="2026-11-02 13:43:20.076131468 +0000 UTC m=+32.869313297"
Nov 02 13:43:21.456305 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.455336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:21.456305 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.455462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:21.456305 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.456011    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:43:21.456305 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.456077    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:43:53.456058532 +0000 UTC m=+66.249240343 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : secret "metrics-daemon-secret" not found
Nov 02 13:43:21.459344 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.459316    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:21.513401 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.513373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:21.556675 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.556648    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:21.559705 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.559680    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:21.792505 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.792425    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:21.860250 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.859416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:21.860250 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.859495    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:21.860250 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.859700    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 13:43:21.860250 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.859752    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls podName:3bfd00d2-35ee-4067-8649-bdb54ee596a2 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:25.859736834 +0000 UTC m=+38.652918640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5ssg4" (UID: "3bfd00d2-35ee-4067-8649-bdb54ee596a2") : secret "samples-operator-tls" not found
Nov 02 13:43:21.860250 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.860171    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:43:21.860250 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.860220    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls podName:63881d4e-a3bf-4948-a369-c892baba23a8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:25.860204916 +0000 UTC m=+38.653386727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls") pod "dns-default-fzlvp" (UID: "63881d4e-a3bf-4948-a369-c892baba23a8") : secret "dns-default-metrics-tls" not found
Nov 02 13:43:21.892368 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.892337    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zp7ql"]
Nov 02 13:43:21.896928 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:21.896898    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00be7ba2_6f06_4ed5_a007_d0ac612177ee.slice/crio-1caa65cb2d0470813f89c06842b8df0d9688ee5799a934234d23476c31c04867 WatchSource:0}: Error finding container 1caa65cb2d0470813f89c06842b8df0d9688ee5799a934234d23476c31c04867: Status 404 returned error can't find the container with id 1caa65cb2d0470813f89c06842b8df0d9688ee5799a934234d23476c31c04867
Nov 02 13:43:21.945517 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.945440    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zp7ql" event={"ID":"00be7ba2-6f06-4ed5-a007-d0ac612177ee","Type":"ContainerStarted","Data":"1caa65cb2d0470813f89c06842b8df0d9688ee5799a934234d23476c31c04867"}
Nov 02 13:43:21.949360 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.948899    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z" event={"ID":"4150923a-78f1-4f52-a0d5-271ec942bd1b","Type":"ContainerStarted","Data":"fc5cf9f210fbfdd191cfc3853c757a8853934b0a2ec0f57f6ae900df7bf2768d"}
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.960115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.960194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.960239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:21.960297    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.960416    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.960430    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d555fdbf5-mnstt: secret "image-registry-tls" not found
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.960479    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls podName:3fa80f89-f24d-4f86-940c-864dbf177a5c nodeName:}" failed. No retries permitted until 2026-11-02 13:43:25.960465822 +0000 UTC m=+38.753647645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls") pod "image-registry-5d555fdbf5-mnstt" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c") : secret "image-registry-tls" not found
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.960848    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:25.96083006 +0000 UTC m=+38.754011886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : configmap references non-existent config key: service-ca.crt
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.960924    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.960958    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:25.960947082 +0000 UTC m=+38.754128901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : secret "router-metrics-certs-default" not found
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.961039    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 13:43:21.961132 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:21.961076    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert podName:b34b2c6f-210f-4256-8383-16f6a3a360bb nodeName:}" failed. No retries permitted until 2026-11-02 13:43:25.961064716 +0000 UTC m=+38.754246537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jc4gv" (UID: "b34b2c6f-210f-4256-8383-16f6a3a360bb") : secret "networking-console-plugin-cert" not found
Nov 02 13:43:22.016623 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:22.015359    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z" podStartSLOduration=341.679744907 podStartE2EDuration="5m44.015330955s" totalImagesPullingTime="2.335586048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:22.015034995 +0000 UTC m=+34.808216823" watchObservedRunningTime="2026-11-02 13:43:22.015330955 +0000 UTC m=+34.808512785"
Nov 02 13:43:22.061700 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:22.061666    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:22.062304 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:22.061876    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:22.062304 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:22.061900    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8c6787bcd-psbdk: secret "image-registry-tls" not found
Nov 02 13:43:22.062304 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:22.061973    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls podName:633c4858-9dc5-4049-aec3-8a14171652ae nodeName:}" failed. No retries permitted until 2026-11-02 13:43:26.061953441 +0000 UTC m=+38.855135261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls") pod "image-registry-8c6787bcd-psbdk" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae") : secret "image-registry-tls" not found
Nov 02 13:43:22.062304 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:22.062068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:22.062304 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:22.062231    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:22.062304 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:22.062262    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls podName:2ca56ffb-997f-4edb-9efa-0609042e8f08 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:26.062253793 +0000 UTC m=+38.855435613 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nnxtg" (UID: "2ca56ffb-997f-4edb-9efa-0609042e8f08") : secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:22.146588 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:22.146143    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2gjsj"]
Nov 02 13:43:22.148425 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:22.148366    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb318d8f_e7de_4185_96cd_d592b304248a.slice/crio-df412506e405502b7f20b0e039f8c2e6cf9c9409107629ce7ad7a3c91aab1928 WatchSource:0}: Error finding container df412506e405502b7f20b0e039f8c2e6cf9c9409107629ce7ad7a3c91aab1928: Status 404 returned error can't find the container with id df412506e405502b7f20b0e039f8c2e6cf9c9409107629ce7ad7a3c91aab1928
Nov 02 13:43:22.956983 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:22.956950    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2gjsj" event={"ID":"fb318d8f-e7de-4185-96cd-d592b304248a","Type":"ContainerStarted","Data":"df412506e405502b7f20b0e039f8c2e6cf9c9409107629ce7ad7a3c91aab1928"}
Nov 02 13:43:23.672957 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.672924    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2m5tg"]
Nov 02 13:43:23.686917 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.686890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.714234 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.713156    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2m5tg"]
Nov 02 13:43:23.724863 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.724801    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9sg29\""
Nov 02 13:43:23.724863 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.724842    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 13:43:23.725106 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.725091    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 13:43:23.750846 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.750824    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-q59bv-5ptkf"]
Nov 02 13:43:23.766824 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.766216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.778899 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.778842    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q59bv-5ptkf"]
Nov 02 13:43:23.781892 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.781871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.782050 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.782023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.782127 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.782086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb92t\" (UniqueName: \"kubernetes.io/projected/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-api-access-hb92t\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.782317 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.782181    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/028f7fde-fe8a-4d88-9c6c-54193c5c490b-data-volume\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.782317 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.782243    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/028f7fde-fe8a-4d88-9c6c-54193c5c490b-crio-socket\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.816957 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.816123    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-n6hr6\""
Nov 02 13:43:23.883294 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.883262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/028f7fde-fe8a-4d88-9c6c-54193c5c490b-crio-socket\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.883447 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.883376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.883447 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.883414    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.883558 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.883468    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.883558 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.883492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.883558 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.883551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.883715 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.883595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb92t\" (UniqueName: \"kubernetes.io/projected/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-api-access-hb92t\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.883715 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.883637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqql\" (UniqueName: \"kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.883715 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.883662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/028f7fde-fe8a-4d88-9c6c-54193c5c490b-data-volume\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.884021 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.884004    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/028f7fde-fe8a-4d88-9c6c-54193c5c490b-data-volume\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.884095 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.884081    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/028f7fde-fe8a-4d88-9c6c-54193c5c490b-crio-socket\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.884210 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:23.884197    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:23.884262 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:23.884254    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:24.384236184 +0000 UTC m=+37.177417996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:23.884560 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:23.884521    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 13:43:23.884633 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:23.884562    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:24.384548934 +0000 UTC m=+37.177730747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : configmap "kube-rbac-proxy" not found
Nov 02 13:43:23.919083 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.919040    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb92t\" (UniqueName: \"kubernetes.io/projected/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-api-access-hb92t\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:23.963293 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.963257    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-f79wg" event={"ID":"8646ac2d-cfd2-4b7c-929a-906815e6551f","Type":"ContainerStarted","Data":"aafbeb05fba56682112c11251207e14667fb860ac195b82aa9d7d61bf56a15a6"}
Nov 02 13:43:23.968217 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.968166    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n" event={"ID":"fb81289b-9a76-4529-a6f5-649ea7c85d67","Type":"ContainerStarted","Data":"5916ca1f7909bd58b5f42ce47aac38ac0f60250d29085c76b17e5b68e1bdda85"}
Nov 02 13:43:23.971344 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.971325    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/0.log"
Nov 02 13:43:23.971459 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.971358    2582 generic.go:356] "Generic (PLEG): container finished" podID="398a3f5c-7cbe-448d-8f16-3360c18a38bf" containerID="f341b992262b9c0b4106e8046a77a14e74b9e597c5de5e5e1b17531c72be5c97" exitCode=255
Nov 02 13:43:23.971459 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.971422    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" event={"ID":"398a3f5c-7cbe-448d-8f16-3360c18a38bf","Type":"ContainerDied","Data":"f341b992262b9c0b4106e8046a77a14e74b9e597c5de5e5e1b17531c72be5c97"}
Nov 02 13:43:23.971673 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.971656    2582 scope.go:122] "RemoveContainer" containerID="f341b992262b9c0b4106e8046a77a14e74b9e597c5de5e5e1b17531c72be5c97"
Nov 02 13:43:23.977066 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.976931    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6" event={"ID":"27e3e0a4-dc29-4e68-9344-f82ea07cb067","Type":"ContainerStarted","Data":"3782015e076b004d734332b23b2bc704dd58e57ef84816ceb3e97e4af27763ee"}
Nov 02 13:43:23.984639 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.984613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.984730 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.984686    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.986284 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.985315    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.986284 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.985415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqql\" (UniqueName: \"kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.986284 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.985549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.986284 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.985566    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:23.986284 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:23.986246    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:24.010804 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.010737    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-f79wg" podStartSLOduration=341.949662293 podStartE2EDuration="5m46.010722987s" totalImagesPullingTime="4.061060694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:24.009894481 +0000 UTC m=+36.803076313" watchObservedRunningTime="2026-11-02 13:43:24.010722987 +0000 UTC m=+36.803904816"
Nov 02 13:43:24.025119 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.025090    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqql\" (UniqueName: \"kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:24.059198 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.058954    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6" podStartSLOduration=338.489611997 podStartE2EDuration="5m43.058935698s" totalImagesPullingTime="4.569323701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:24.057085673 +0000 UTC m=+36.850267502" watchObservedRunningTime="2026-11-02 13:43:24.058935698 +0000 UTC m=+36.852117526"
Nov 02 13:43:24.097440 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.097414    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:24.103103 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.103053    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n" podStartSLOduration=336.403395019 podStartE2EDuration="5m41.103033204s" totalImagesPullingTime="4.699638185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:24.101562481 +0000 UTC m=+36.894744311" watchObservedRunningTime="2026-11-02 13:43:24.103033204 +0000 UTC m=+36.896215034"
Nov 02 13:43:24.393837 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.390610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:24.393837 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:24.390743    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 13:43:24.393837 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:24.390836    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:25.390814369 +0000 UTC m=+38.183996179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : configmap "kube-rbac-proxy" not found
Nov 02 13:43:24.393837 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.390811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:24.393837 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:24.390923    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:24.393837 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:24.390972    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:25.390958894 +0000 UTC m=+38.184140718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:24.459310 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.459285    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q59bv-5ptkf"]
Nov 02 13:43:24.461643 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:24.461614    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99fc1fe5_048b_4196_b66c_771710cd73c0.slice/crio-6cd201012c26fed6351658b222ea0788f7f9813fcc30d7408e37e40c4b6fa864 WatchSource:0}: Error finding container 6cd201012c26fed6351658b222ea0788f7f9813fcc30d7408e37e40c4b6fa864: Status 404 returned error can't find the container with id 6cd201012c26fed6351658b222ea0788f7f9813fcc30d7408e37e40c4b6fa864
Nov 02 13:43:24.549806 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.548538    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"]
Nov 02 13:43:24.573807 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.572699    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"
Nov 02 13:43:24.573807 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.573602    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"]
Nov 02 13:43:24.582600 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.582578    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"]
Nov 02 13:43:24.582710 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.582693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"
Nov 02 13:43:24.614648 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.614626    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ffh9b\""
Nov 02 13:43:24.617251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.616930    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 13:43:24.617251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.616953    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:24.617377 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.617357    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"]
Nov 02 13:43:24.693847 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.693727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhqc\" (UniqueName: \"kubernetes.io/projected/1859de72-db9b-4e60-ac47-b8360156ed1a-kube-api-access-qfhqc\") pod \"migrator-66df5d496b-697rr\" (UID: \"1859de72-db9b-4e60-ac47-b8360156ed1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"
Nov 02 13:43:24.694181 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.693998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhb7\" (UniqueName: \"kubernetes.io/projected/ea36ca35-7e60-41a3-b25a-0e39965f22bf-kube-api-access-srhb7\") pod \"migrator-66df5d496b-wn9j4\" (UID: \"ea36ca35-7e60-41a3-b25a-0e39965f22bf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"
Nov 02 13:43:24.795737 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.795294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhqc\" (UniqueName: \"kubernetes.io/projected/1859de72-db9b-4e60-ac47-b8360156ed1a-kube-api-access-qfhqc\") pod \"migrator-66df5d496b-697rr\" (UID: \"1859de72-db9b-4e60-ac47-b8360156ed1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"
Nov 02 13:43:24.795737 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.795381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhb7\" (UniqueName: \"kubernetes.io/projected/ea36ca35-7e60-41a3-b25a-0e39965f22bf-kube-api-access-srhb7\") pod \"migrator-66df5d496b-wn9j4\" (UID: \"ea36ca35-7e60-41a3-b25a-0e39965f22bf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"
Nov 02 13:43:24.826244 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.826173    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhqc\" (UniqueName: \"kubernetes.io/projected/1859de72-db9b-4e60-ac47-b8360156ed1a-kube-api-access-qfhqc\") pod \"migrator-66df5d496b-697rr\" (UID: \"1859de72-db9b-4e60-ac47-b8360156ed1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"
Nov 02 13:43:24.830637 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.830547    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhb7\" (UniqueName: \"kubernetes.io/projected/ea36ca35-7e60-41a3-b25a-0e39965f22bf-kube-api-access-srhb7\") pod \"migrator-66df5d496b-wn9j4\" (UID: \"ea36ca35-7e60-41a3-b25a-0e39965f22bf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"
Nov 02 13:43:24.911457 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.911398    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"
Nov 02 13:43:24.922711 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:24.922680    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"
Nov 02 13:43:25.004517 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.004489    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/1.log"
Nov 02 13:43:25.007951 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.006104    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/0.log"
Nov 02 13:43:25.007951 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.006143    2582 generic.go:356] "Generic (PLEG): container finished" podID="398a3f5c-7cbe-448d-8f16-3360c18a38bf" containerID="0c97ab36897f8243ae5de4891c4454237119719e8b9ba48f3e20f4dddeb9ce3f" exitCode=255
Nov 02 13:43:25.007951 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.006240    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" event={"ID":"398a3f5c-7cbe-448d-8f16-3360c18a38bf","Type":"ContainerDied","Data":"0c97ab36897f8243ae5de4891c4454237119719e8b9ba48f3e20f4dddeb9ce3f"}
Nov 02 13:43:25.007951 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.006327    2582 scope.go:122] "RemoveContainer" containerID="f341b992262b9c0b4106e8046a77a14e74b9e597c5de5e5e1b17531c72be5c97"
Nov 02 13:43:25.007951 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.006732    2582 scope.go:122] "RemoveContainer" containerID="0c97ab36897f8243ae5de4891c4454237119719e8b9ba48f3e20f4dddeb9ce3f"
Nov 02 13:43:25.007951 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:25.006958    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zfzrk_openshift-console-operator(398a3f5c-7cbe-448d-8f16-3360c18a38bf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" podUID="398a3f5c-7cbe-448d-8f16-3360c18a38bf"
Nov 02 13:43:25.024300 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.023587    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q59bv-5ptkf" event={"ID":"99fc1fe5-048b-4196-b66c-771710cd73c0","Type":"ContainerStarted","Data":"c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b"}
Nov 02 13:43:25.024300 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.023664    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q59bv-5ptkf" event={"ID":"99fc1fe5-048b-4196-b66c-771710cd73c0","Type":"ContainerStarted","Data":"6cd201012c26fed6351658b222ea0788f7f9813fcc30d7408e37e40c4b6fa864"}
Nov 02 13:43:25.033032 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.031801    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7nn8" event={"ID":"df16fea1-6bca-47f3-b9bd-35716e3d50f3","Type":"ContainerStarted","Data":"74861c55dbf720ac5b4d002fddfe97fe8ec3532e605c5cefd646a1657c88ccc5"}
Nov 02 13:43:25.106157 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.106107    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b7nn8" podStartSLOduration=1.631497527 podStartE2EDuration="7.106089491s" totalImagesPullingTime="5.474591964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:25.105328555 +0000 UTC m=+37.898510385" watchObservedRunningTime="2026-11-02 13:43:25.106089491 +0000 UTC m=+37.899271320"
Nov 02 13:43:25.149748 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.149646    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-q59bv-5ptkf" podStartSLOduration=2.149627741 podStartE2EDuration="2.149627741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:25.149329304 +0000 UTC m=+37.942511130" watchObservedRunningTime="2026-11-02 13:43:25.149627741 +0000 UTC m=+37.942809570"
Nov 02 13:43:25.240179 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.240152    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"]
Nov 02 13:43:25.241556 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:25.241510    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1859de72_db9b_4e60_ac47_b8360156ed1a.slice/crio-0e5bc3a3ddad83a4131da521da3db8121936be915c02f2ef1ab89ac06c5df0f8 WatchSource:0}: Error finding container 0e5bc3a3ddad83a4131da521da3db8121936be915c02f2ef1ab89ac06c5df0f8: Status 404 returned error can't find the container with id 0e5bc3a3ddad83a4131da521da3db8121936be915c02f2ef1ab89ac06c5df0f8
Nov 02 13:43:25.280490 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.280463    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"]
Nov 02 13:43:25.280888 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:25.280862    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea36ca35_7e60_41a3_b25a_0e39965f22bf.slice/crio-5215d070599282003b08c100539b8ffd414603538e2834720bbc7b12979aeb9a WatchSource:0}: Error finding container 5215d070599282003b08c100539b8ffd414603538e2834720bbc7b12979aeb9a: Status 404 returned error can't find the container with id 5215d070599282003b08c100539b8ffd414603538e2834720bbc7b12979aeb9a
Nov 02 13:43:25.407867 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.407831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:25.408032 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.407932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:25.408032 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:25.407991    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:25.408152 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:25.408058    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:27.408038535 +0000 UTC m=+40.201220358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:25.408152 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:25.408096    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 13:43:25.408152 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:25.408134    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:27.408122033 +0000 UTC m=+40.201303845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : configmap "kube-rbac-proxy" not found
Nov 02 13:43:25.414543 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.414516    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cggsz"]
Nov 02 13:43:25.441120 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.441094    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cggsz"]
Nov 02 13:43:25.442253 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.441208    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.458675 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.458652    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 13:43:25.458902 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.458879    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 13:43:25.458902 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.458898    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 13:43:25.459138 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.459124    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 13:43:25.459209 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.459195    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jws44\""
Nov 02 13:43:25.609460 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.609389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fv5\" (UniqueName: \"kubernetes.io/projected/0fb4e264-b7b8-46de-b222-25331e6400c4-kube-api-access-g4fv5\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.609588 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.609543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-cabundle\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.609688 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.609671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-key\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.710654 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.710623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-key\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.710814 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.710751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fv5\" (UniqueName: \"kubernetes.io/projected/0fb4e264-b7b8-46de-b222-25331e6400c4-kube-api-access-g4fv5\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.710885 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.710867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-cabundle\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.711699 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.711675    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-cabundle\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.713258 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.713237    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-key\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.739172 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.739140    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fv5\" (UniqueName: \"kubernetes.io/projected/0fb4e264-b7b8-46de-b222-25331e6400c4-kube-api-access-g4fv5\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.778278 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.778251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:25.913289 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.913223    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:25.913410 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:25.913306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:25.913519 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:25.913503    2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 02 13:43:25.913578 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:25.913569    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls podName:3bfd00d2-35ee-4067-8649-bdb54ee596a2 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:33.91355049 +0000 UTC m=+46.706732312 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5ssg4" (UID: "3bfd00d2-35ee-4067-8649-bdb54ee596a2") : secret "samples-operator-tls" not found
Nov 02 13:43:25.913954 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:25.913939    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:43:25.914000 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:25.913987    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls podName:63881d4e-a3bf-4948-a369-c892baba23a8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:33.913976133 +0000 UTC m=+46.707157951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls") pod "dns-default-fzlvp" (UID: "63881d4e-a3bf-4948-a369-c892baba23a8") : secret "dns-default-metrics-tls" not found
Nov 02 13:43:26.014299 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.014271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.014372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.014458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.014496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.014617    2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.014666    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert podName:b34b2c6f-210f-4256-8383-16f6a3a360bb nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.014653502 +0000 UTC m=+46.807835323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jc4gv" (UID: "b34b2c6f-210f-4256-8383-16f6a3a360bb") : secret "networking-console-plugin-cert" not found
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.015117    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.015134    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5d555fdbf5-mnstt: secret "image-registry-tls" not found
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.015180    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls podName:3fa80f89-f24d-4f86-940c-864dbf177a5c nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.015165669 +0000 UTC m=+46.808347489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls") pod "image-registry-5d555fdbf5-mnstt" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c") : secret "image-registry-tls" not found
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.015250    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.015241463 +0000 UTC m=+46.808423283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : configmap references non-existent config key: service-ca.crt
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.015310    2582 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 02 13:43:26.016961 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.015338    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.015328517 +0000 UTC m=+46.808510339 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : secret "router-metrics-certs-default" not found
Nov 02 13:43:26.049055 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.045848    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4" event={"ID":"ea36ca35-7e60-41a3-b25a-0e39965f22bf","Type":"ContainerStarted","Data":"5215d070599282003b08c100539b8ffd414603538e2834720bbc7b12979aeb9a"}
Nov 02 13:43:26.049055 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.048734    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr" event={"ID":"1859de72-db9b-4e60-ac47-b8360156ed1a","Type":"ContainerStarted","Data":"0e5bc3a3ddad83a4131da521da3db8121936be915c02f2ef1ab89ac06c5df0f8"}
Nov 02 13:43:26.053431 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.053410    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/1.log"
Nov 02 13:43:26.054251 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.054234    2582 scope.go:122] "RemoveContainer" containerID="0c97ab36897f8243ae5de4891c4454237119719e8b9ba48f3e20f4dddeb9ce3f"
Nov 02 13:43:26.054436 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.054414    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zfzrk_openshift-console-operator(398a3f5c-7cbe-448d-8f16-3360c18a38bf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" podUID="398a3f5c-7cbe-448d-8f16-3360c18a38bf"
Nov 02 13:43:26.115076 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.115041    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:26.115199 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.115167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:26.115199 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.115185    2582 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:26.115307 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.115238    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls podName:2ca56ffb-997f-4edb-9efa-0609042e8f08 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.115222188 +0000 UTC m=+46.908404009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-nnxtg" (UID: "2ca56ffb-997f-4edb-9efa-0609042e8f08") : secret "cluster-monitoring-operator-tls" not found
Nov 02 13:43:26.115307 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.115293    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 02 13:43:26.115307 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.115302    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8c6787bcd-psbdk: secret "image-registry-tls" not found
Nov 02 13:43:26.115454 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:26.115333    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls podName:633c4858-9dc5-4049-aec3-8a14171652ae nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.11532547 +0000 UTC m=+46.908507276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls") pod "image-registry-8c6787bcd-psbdk" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae") : secret "image-registry-tls" not found
Nov 02 13:43:26.284523 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:26.284484    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cggsz"]
Nov 02 13:43:26.287408 ip-10-0-139-26 kubenswrapper[2582]: W1102 13:43:26.287376    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb4e264_b7b8_46de_b222_25331e6400c4.slice/crio-9e1fa3d5c174e3f7eed0f34a4b335339ccfa8d91af01c486404663cbfac1247b WatchSource:0}: Error finding container 9e1fa3d5c174e3f7eed0f34a4b335339ccfa8d91af01c486404663cbfac1247b: Status 404 returned error can't find the container with id 9e1fa3d5c174e3f7eed0f34a4b335339ccfa8d91af01c486404663cbfac1247b
Nov 02 13:43:27.059448 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.059324    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5" event={"ID":"488bddc1-06e8-446c-b612-5ddeac1d8bdf","Type":"ContainerStarted","Data":"a33789b87da766c06920b7f32bd87a860c04d97764cef12a84d689e12e531a31"}
Nov 02 13:43:27.061156 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.061125    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2gjsj" event={"ID":"fb318d8f-e7de-4185-96cd-d592b304248a","Type":"ContainerStarted","Data":"e57abf8ef6d092c7d8d150487c00aad616e3fc04eb8e1765b44061631dba16c2"}
Nov 02 13:43:27.061296 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.061196    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:27.062716 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.062691    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz" event={"ID":"0fb4e264-b7b8-46de-b222-25331e6400c4","Type":"ContainerStarted","Data":"584c9bb4bc63c6697eca0b0daeb2a36d6a6f4458db1d10def9fd81703a2051db"}
Nov 02 13:43:27.062917 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.062729    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz" event={"ID":"0fb4e264-b7b8-46de-b222-25331e6400c4","Type":"ContainerStarted","Data":"9e1fa3d5c174e3f7eed0f34a4b335339ccfa8d91af01c486404663cbfac1247b"}
Nov 02 13:43:27.107358 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.107008    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5" podStartSLOduration=340.130389373 podStartE2EDuration="5m47.106995737s" totalImagesPullingTime="6.976606364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:27.106456599 +0000 UTC m=+39.899638431" watchObservedRunningTime="2026-11-02 13:43:27.106995737 +0000 UTC m=+39.900177565"
Nov 02 13:43:27.153686 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.152186    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2gjsj" podStartSLOduration=34.331323249 podStartE2EDuration="38.152169534s" totalImagesPullingTime="3.820846285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:27.151731528 +0000 UTC m=+39.944913357" watchObservedRunningTime="2026-11-02 13:43:27.152169534 +0000 UTC m=+39.945351363"
Nov 02 13:43:27.200732 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.200690    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz" podStartSLOduration=2.200677245 podStartE2EDuration="2.200677245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:27.200487645 +0000 UTC m=+39.993669476" watchObservedRunningTime="2026-11-02 13:43:27.200677245 +0000 UTC m=+39.993859087"
Nov 02 13:43:27.429625 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.429590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:27.429807 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:27.429699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:27.429807 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:27.429765    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:27.429929 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:27.429817    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 13:43:27.429929 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:27.429866    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:31.429843816 +0000 UTC m=+44.223025635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:27.429929 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:27.429889    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:31.429878004 +0000 UTC m=+44.223059811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : configmap "kube-rbac-proxy" not found
Nov 02 13:43:28.068343 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:28.068090    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4" event={"ID":"ea36ca35-7e60-41a3-b25a-0e39965f22bf","Type":"ContainerStarted","Data":"540f0bb2f6855bd2cc91369563988a2ebac885eb989d04260b241029f82ceb05"}
Nov 02 13:43:28.069937 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:28.069907    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr" event={"ID":"1859de72-db9b-4e60-ac47-b8360156ed1a","Type":"ContainerStarted","Data":"b0286f2882c2bc50b4fa5c479b4be6fea0adfdbf50b2d9affc49b065fabaffa1"}
Nov 02 13:43:28.120511 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:28.120450    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4" podStartSLOduration=1.8644280869999998 podStartE2EDuration="4.120433205s" totalImagesPullingTime="2.256005118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:28.120077886 +0000 UTC m=+40.913259718" watchObservedRunningTime="2026-11-02 13:43:28.120433205 +0000 UTC m=+40.913615033"
Nov 02 13:43:28.168593 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:28.168541    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr" podStartSLOduration=2.156743625 podStartE2EDuration="4.168522972s" totalImagesPullingTime="2.011779347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:28.167475196 +0000 UTC m=+40.960657023" watchObservedRunningTime="2026-11-02 13:43:28.168522972 +0000 UTC m=+40.961704804"
Nov 02 13:43:28.394869 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:28.394802    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:28.395007 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:28.394872    2582 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:28.395262 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:28.395248    2582 scope.go:122] "RemoveContainer" containerID="0c97ab36897f8243ae5de4891c4454237119719e8b9ba48f3e20f4dddeb9ce3f"
Nov 02 13:43:28.395495 ip-10-0-139-26 kubenswrapper[2582]: E1102 13:43:28.395452    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zfzrk_openshift-console-operator(398a3f5c-7cbe-448d-8f16-3360c18a38bf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" podUID="398a3f5c-7cbe-448d-8f16-3360c18a38bf"
Nov 02 13:43:29.076906 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:29.076867    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zp7ql" event={"ID":"00be7ba2-6f06-4ed5-a007-d0ac612177ee","Type":"ContainerStarted","Data":"edcdac0ca15e4a4fc9fb130297e1d73dd7c21e115b87d8c08262782eb502c674"}
Nov 02 13:43:29.135172 ip-10-0-139-26 kubenswrapper[2582]: I1102 13:43:29.135127    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zp7ql" podStartSLOduration=33.138648973 podStartE2EDuration="40.135114734s" totalImagesPullingTime="6.996465761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 13:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:29.134910582 +0000 UTC m=+41.928092410" watchObservedRunningTime="2026-11-02 13:43:29.135114734 +0000 UTC m=+41.928296561"
Nov 02 13:43:29.188976 ip-10-0-139-26 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 13:43:29.203187 ip-10-0-139-26 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 13:43:29.203353 ip-10-0-139-26 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 13:43:29.203404 ip-10-0-139-26 systemd[1]: kubelet.service: Consumed 3.035s CPU time.
Nov 02 13:43:29.218259 ip-10-0-139-26 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 13:43:29.305676 ip-10-0-139-26 kubenswrapper[5034]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:43:29.305676 ip-10-0-139-26 kubenswrapper[5034]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 13:43:29.305676 ip-10-0-139-26 kubenswrapper[5034]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:43:29.305676 ip-10-0-139-26 kubenswrapper[5034]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 13:43:29.307848 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307759    5034 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:29.307848 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307847    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:29.307848 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307851    5034 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307854    5034 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307858    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307861    5034 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307872    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307876    5034 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307879    5034 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307882    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307885    5034 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307888    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307890    5034 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307893    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307895    5034 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307898    5034 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307901    5034 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307904    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307907    5034 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307910    5034 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307913    5034 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:29.307951 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307917    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307921    5034 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307925    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307929    5034 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307932    5034 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307935    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307938    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307940    5034 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307944    5034 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307947    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307949    5034 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307952    5034 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307955    5034 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307958    5034 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307960    5034 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307963    5034 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307966    5034 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307969    5034 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307972    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307975    5034 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:29.308412 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307978    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307980    5034 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307983    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307986    5034 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307988    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307991    5034 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307993    5034 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.307998    5034 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308002    5034 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308005    5034 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308008    5034 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308010    5034 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308013    5034 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308016    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308019    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308022    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308025    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308027    5034 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308030    5034 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:29.308928 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308033    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308035    5034 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308038    5034 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308041    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308043    5034 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308046    5034 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308048    5034 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308051    5034 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308054    5034 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308056    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308059    5034 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308061    5034 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308064    5034 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308067    5034 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308069    5034 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308072    5034 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308074    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308077    5034 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308080    5034 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308082    5034 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:29.309376 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308085    5034 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308087    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308090    5034 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308092    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308095    5034 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308098    5034 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308100    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308103    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308106    5034 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308109    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308111    5034 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308114    5034 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308116    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308119    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308122    5034 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308124    5034 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308127    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308131    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308133    5034 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308136    5034 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:29.309857 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308138    5034 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308141    5034 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308143    5034 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308146    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308148    5034 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308152    5034 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308948    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308956    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308960    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308963    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308966    5034 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308968    5034 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308971    5034 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308975    5034 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308979    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308982    5034 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308985    5034 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308988    5034 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308991    5034 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:29.310382 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308994    5034 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308997    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.308999    5034 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309002    5034 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309004    5034 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309007    5034 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309009    5034 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309012    5034 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309015    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309017    5034 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309020    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309023    5034 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309026    5034 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309029    5034 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309031    5034 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309034    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309036    5034 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309039    5034 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309042    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:29.310867 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309045    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309048    5034 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309050    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309053    5034 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309056    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309058    5034 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309061    5034 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309065    5034 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309067    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309070    5034 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309073    5034 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309075    5034 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309078    5034 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309081    5034 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309083    5034 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309086    5034 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309088    5034 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309091    5034 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309093    5034 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:29.311554 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309097    5034 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309099    5034 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309101    5034 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309104    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309106    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309109    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309111    5034 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309114    5034 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309117    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309119    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309122    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309124    5034 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309127    5034 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309130    5034 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309133    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309137    5034 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309141    5034 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309143    5034 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309157    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:29.312048 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309162    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309166    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309169    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309172    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309175    5034 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309178    5034 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309180    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309183    5034 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309186    5034 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309188    5034 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309191    5034 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309193    5034 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309196    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309198    5034 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309201    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309203    5034 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309206    5034 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309209    5034 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309211    5034 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309214    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:29.312521 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309217    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309220    5034 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309222    5034 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309225    5034 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309227    5034 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309230    5034 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309232    5034 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309235    5034 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309238    5034 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309241    5034 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309243    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309246    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309249    5034 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309252    5034 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309255    5034 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309257    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309756    5034 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309762    5034 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309766    5034 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309769    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:29.313044 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309773    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309790    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309793    5034 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309796    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309800    5034 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309802    5034 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309805    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309808    5034 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309810    5034 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309813    5034 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309816    5034 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309818    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309821    5034 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309823    5034 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309826    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309829    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309831    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309833    5034 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309836    5034 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309839    5034 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:29.313523 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309848    5034 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309851    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309853    5034 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309856    5034 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309859    5034 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309862    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309865    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309868    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309870    5034 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309873    5034 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309876    5034 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309878    5034 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309881    5034 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309883    5034 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309887    5034 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309891    5034 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309894    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309896    5034 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309899    5034 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:29.314060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309901    5034 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309904    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309906    5034 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309909    5034 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309911    5034 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309914    5034 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309917    5034 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309919    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309922    5034 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309924    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309927    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309929    5034 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309932    5034 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309934    5034 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309942    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309945    5034 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309951    5034 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309953    5034 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309956    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309958    5034 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:29.314503 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309961    5034 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309963    5034 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309966    5034 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309968    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309971    5034 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309973    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309976    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309979    5034 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309981    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309984    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309988    5034 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309992    5034 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309997    5034 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.309999    5034 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310002    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310004    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310007    5034 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310010    5034 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310012    5034 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:29.315027 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310016    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310018    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310021    5034 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310023    5034 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310026    5034 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310029    5034 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310031    5034 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310034    5034 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310040    5034 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310044    5034 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310047    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310049    5034 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310052    5034 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310055    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310058    5034 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310060    5034 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310063    5034 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310065    5034 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310068    5034 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:29.315633 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310071    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310073    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310075    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310078    5034 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310080    5034 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310183    5034 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310192    5034 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310198    5034 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310203    5034 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310207    5034 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310210    5034 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310214    5034 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310218    5034 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310222    5034 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310225    5034 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310228    5034 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310232    5034 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310235    5034 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310238    5034 flags.go:64] FLAG: --cgroup-root=""
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310241    5034 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310244    5034 flags.go:64] FLAG: --client-ca-file=""
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310247    5034 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310255    5034 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310259    5034 flags.go:64] FLAG: --cluster-domain=""
Nov 02 13:43:29.316402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310261    5034 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310265    5034 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310268    5034 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310272    5034 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310282    5034 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310285    5034 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310288    5034 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310291    5034 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310294    5034 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310297    5034 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310300    5034 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310303    5034 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310306    5034 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310311    5034 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310316    5034 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310319    5034 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310322    5034 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310325    5034 flags.go:64] FLAG: --enable-server="true"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310328    5034 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310332    5034 flags.go:64] FLAG: --event-burst="100"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310335    5034 flags.go:64] FLAG: --event-qps="50"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310338    5034 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310341    5034 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310344    5034 flags.go:64] FLAG: --eviction-hard=""
Nov 02 13:43:29.317079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310348    5034 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310351    5034 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310354    5034 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310357    5034 flags.go:64] FLAG: --eviction-soft=""
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310360    5034 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310363    5034 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310368    5034 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310376    5034 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310380    5034 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310383    5034 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310386    5034 flags.go:64] FLAG: --feature-gates=""
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310390    5034 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310394    5034 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310397    5034 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310400    5034 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310403    5034 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310406    5034 flags.go:64] FLAG: --help="false"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310409    5034 flags.go:64] FLAG: --hostname-override="ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310412    5034 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310415    5034 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310418    5034 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310422    5034 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310427    5034 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310430    5034 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 13:43:29.317674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310432    5034 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310435    5034 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310438    5034 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310441    5034 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310444    5034 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310447    5034 flags.go:64] FLAG: --kube-reserved=""
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310450    5034 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310454    5034 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310457    5034 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310460    5034 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310463    5034 flags.go:64] FLAG: --lock-file=""
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310465    5034 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310468    5034 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310472    5034 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310479    5034 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310482    5034 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310485    5034 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310488    5034 flags.go:64] FLAG: --logging-format="text"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310491    5034 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310495    5034 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310498    5034 flags.go:64] FLAG: --manifest-url=""
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310501    5034 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310505    5034 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310509    5034 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310512    5034 flags.go:64] FLAG: --max-pods="110"
Nov 02 13:43:29.318335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310515    5034 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310518    5034 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310522    5034 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310525    5034 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310527    5034 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310532    5034 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310535    5034 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310542    5034 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310545    5034 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310548    5034 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310552    5034 flags.go:64] FLAG: --pod-cidr=""
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310555    5034 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310558    5034 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310561    5034 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310564    5034 flags.go:64] FLAG: --port="10250"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310566    5034 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310569    5034 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09f42ae1c1b5205ce"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310573    5034 flags.go:64] FLAG: --qos-reserved=""
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310576    5034 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310579    5034 flags.go:64] FLAG: --register-node="true"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310582    5034 flags.go:64] FLAG: --register-with-taints=""
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310588    5034 flags.go:64] FLAG: --registry-burst="10"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310591    5034 flags.go:64] FLAG: --registry-qps="5"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310594    5034 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310598    5034 flags.go:64] FLAG: --reserved-memory=""
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310602    5034 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 13:43:29.319023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310605    5034 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310608    5034 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310611    5034 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310614    5034 flags.go:64] FLAG: --runonce="false"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310617    5034 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310620    5034 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310623    5034 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310626    5034 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310629    5034 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310632    5034 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310635    5034 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310639    5034 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310642    5034 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310645    5034 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310648    5034 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310651    5034 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310654    5034 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310657    5034 flags.go:64] FLAG: --system-cgroups=""
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310659    5034 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310665    5034 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310668    5034 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310671    5034 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310677    5034 flags.go:64] FLAG: --tls-min-version=""
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310681    5034 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310683    5034 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 13:43:29.319642 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310686    5034 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310690    5034 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310694    5034 flags.go:64] FLAG: --v="2"
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310698    5034 flags.go:64] FLAG: --version="false"
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310703    5034 flags.go:64] FLAG: --vmodule=""
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310707    5034 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.310713    5034 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310931    5034 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310941    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310946    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310950    5034 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310954    5034 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310958    5034 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310963    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310968    5034 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310972    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310976    5034 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310990    5034 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.310997    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311002    5034 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311006    5034 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:29.320247 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311010    5034 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311013    5034 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311016    5034 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311018    5034 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311021    5034 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311024    5034 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311027    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311029    5034 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311032    5034 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311034    5034 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311037    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311040    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311043    5034 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311047    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311049    5034 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311052    5034 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311055    5034 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311057    5034 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311069    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311072    5034 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:29.320763 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311075    5034 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311078    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311080    5034 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311083    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311087    5034 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311091    5034 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311094    5034 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311097    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311100    5034 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311108    5034 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311111    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311114    5034 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311116    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311119    5034 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311121    5034 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311124    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311127    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311130    5034 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311132    5034 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:29.321367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311135    5034 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311138    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311140    5034 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311143    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311145    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311148    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311152    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311155    5034 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311157    5034 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311160    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311163    5034 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311165    5034 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311169    5034 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311172    5034 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311174    5034 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311177    5034 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311179    5034 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311182    5034 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311185    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311187    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:29.321949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311189    5034 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311192    5034 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311197    5034 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311199    5034 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311202    5034 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311204    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311207    5034 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311210    5034 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311212    5034 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311214    5034 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311217    5034 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311219    5034 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311222    5034 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311225    5034 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311227    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311229    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311232    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311234    5034 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311238    5034 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:29.322545 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311241    5034 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311244    5034 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311246    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311249    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311252    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311254    5034 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311257    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311260    5034 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311263    5034 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311265    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311268    5034 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311270    5034 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311273    5034 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311277    5034 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.311306    5034 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:43:29.323042 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311358    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311362    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311365    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311368    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311371    5034 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311374    5034 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311376    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311379    5034 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311382    5034 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311384    5034 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311387    5034 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311390    5034 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311393    5034 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311395    5034 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311398    5034 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311401    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311403    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311407    5034 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311410    5034 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311413    5034 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:29.323452 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311416    5034 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311418    5034 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311421    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311424    5034 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311427    5034 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311430    5034 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311433    5034 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311435    5034 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311438    5034 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311440    5034 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311443    5034 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311445    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311450    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311452    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311455    5034 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311458    5034 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311460    5034 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311463    5034 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:29.324067 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311465    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311468    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311470    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311473    5034 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311476    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311478    5034 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311481    5034 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311483    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311486    5034 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311488    5034 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311491    5034 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311496    5034 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311500    5034 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311504    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311507    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311510    5034 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311512    5034 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311515    5034 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311519    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:29.324637 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311527    5034 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311530    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311533    5034 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311535    5034 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311538    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311540    5034 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311543    5034 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311547    5034 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311550    5034 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311552    5034 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311555    5034 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311558    5034 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311561    5034 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311564    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311566    5034 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311569    5034 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311572    5034 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311575    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311578    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311580    5034 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:29.325145 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311583    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311585    5034 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311588    5034 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311591    5034 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311594    5034 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311597    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311599    5034 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311602    5034 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311604    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311607    5034 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311609    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311613    5034 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311616    5034 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311625    5034 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311628    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311631    5034 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311633    5034 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311636    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311638    5034 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311642    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:29.325640 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311645    5034 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311647    5034 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311650    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311652    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311655    5034 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311657    5034 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311661    5034 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311663    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.311666    5034 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.315879    5034 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.315899    5034 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.315921    5034 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.315929    5034 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316101    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316109    5034 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316114    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316119    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316125    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316131    5034 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:29.326142 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316141    5034 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316146    5034 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316151    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316156    5034 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316162    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316166    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316170    5034 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316175    5034 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316179    5034 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316183    5034 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316188    5034 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316192    5034 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316202    5034 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316206    5034 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316210    5034 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316215    5034 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316220    5034 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316224    5034 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316228    5034 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316233    5034 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:29.326638 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316237    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316242    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316246    5034 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316251    5034 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316256    5034 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316265    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316270    5034 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316274    5034 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316278    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316282    5034 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316287    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316291    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316296    5034 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316300    5034 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316305    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316309    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316313    5034 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316317    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316326    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316330    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:29.327262 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316334    5034 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316339    5034 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316343    5034 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316347    5034 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316352    5034 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316356    5034 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316360    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316364    5034 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316368    5034 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316372    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316380    5034 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316385    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316389    5034 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316393    5034 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316397    5034 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316401    5034 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316406    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316411    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316416    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316420    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:29.327738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316425    5034 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316429    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316433    5034 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316441    5034 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316445    5034 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316448    5034 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316451    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316454    5034 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316456    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316459    5034 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316463    5034 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316467    5034 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316470    5034 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316474    5034 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316477    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316486    5034 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316490    5034 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316495    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316500    5034 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:29.328216 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316522    5034 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316525    5034 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316528    5034 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316533    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316625    5034 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316628    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316631    5034 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316634    5034 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316639    5034 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316642    5034 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316653    5034 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316655    5034 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316658    5034 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316661    5034 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316664    5034 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316666    5034 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316669    5034 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316671    5034 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316674    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:29.328915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316676    5034 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316679    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.316710    5034 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316866    5034 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316872    5034 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316876    5034 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316879    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316882    5034 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316890    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316893    5034 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316896    5034 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316898    5034 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316901    5034 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316904    5034 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316906    5034 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316910    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:29.329367 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316914    5034 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316918    5034 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316922    5034 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316926    5034 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316930    5034 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316934    5034 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316938    5034 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316941    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316943    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316946    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316948    5034 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316951    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316954    5034 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316956    5034 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316960    5034 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316965    5034 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316968    5034 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316970    5034 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316973    5034 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:29.329800 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316976    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316978    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316981    5034 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316984    5034 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316986    5034 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316989    5034 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316993    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316996    5034 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.316999    5034 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317001    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317004    5034 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317006    5034 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317009    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317011    5034 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317014    5034 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317016    5034 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317019    5034 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317022    5034 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317024    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:29.330410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317027    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317030    5034 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317032    5034 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317035    5034 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317037    5034 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317040    5034 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317042    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317045    5034 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317048    5034 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317051    5034 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317054    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317057    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317060    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317062    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317065    5034 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317068    5034 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317070    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317073    5034 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317075    5034 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:29.331139 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317078    5034 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317081    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317084    5034 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317087    5034 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317090    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317092    5034 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317095    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317098    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317100    5034 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317105    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317108    5034 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317110    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317113    5034 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317115    5034 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317119    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317122    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317125    5034 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317127    5034 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317130    5034 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317133    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:29.331915 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317135    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317138    5034 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317141    5034 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317143    5034 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317146    5034 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317148    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317151    5034 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317154    5034 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317156    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317159    5034 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317162    5034 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317164    5034 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317167    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317169    5034 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317173    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:29.332658 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317175    5034 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.317200    5034 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317233    5034 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317237    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317240    5034 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317243    5034 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317247    5034 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317250    5034 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317253    5034 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317255    5034 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317258    5034 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317261    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317266    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317269    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317271    5034 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317274    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 13:43:29.333410 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317277    5034 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317279    5034 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317283    5034 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317287    5034 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317290    5034 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317293    5034 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317296    5034 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317299    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317301    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317304    5034 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317307    5034 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317310    5034 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317312    5034 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317315    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317318    5034 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317320    5034 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317324    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317327    5034 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317330    5034 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 13:43:29.334155 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317332    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317335    5034 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317337    5034 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317341    5034 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317344    5034 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317346    5034 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317349    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317351    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317354    5034 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317358    5034 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317360    5034 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317363    5034 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317365    5034 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317369    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317371    5034 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317375    5034 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317379    5034 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317383    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317386    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 13:43:29.335239 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317388    5034 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317391    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317393    5034 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317396    5034 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317400    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317402    5034 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317405    5034 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317408    5034 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317410    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317413    5034 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317416    5034 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317419    5034 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317422    5034 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317424    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317426    5034 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317429    5034 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317433    5034 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317436    5034 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317438    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317441    5034 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 13:43:29.336192 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317443    5034 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317446    5034 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317450    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317452    5034 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317455    5034 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317458    5034 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317460    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317463    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317465    5034 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317468    5034 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317470    5034 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317473    5034 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317475    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317478    5034 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317481    5034 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317484    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317486    5034 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317489    5034 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317492    5034 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 13:43:29.337059 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317494    5034 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317497    5034 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317501    5034 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317503    5034 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317506    5034 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317509    5034 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317511    5034 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317514    5034 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317516    5034 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317520    5034 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317523    5034 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317526    5034 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317528    5034 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317531    5034 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:29.317533    5034 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.317605    5034 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.318555    5034 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.318607    5034 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.318797    5034 server.go:1014] "Starting client certificate rotation"
Nov 02 13:43:29.337888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.318934    5034 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 13:43:29.338377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.319010    5034 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 13:37:48 +0000 UTC" deadline="2028-05-10 21:30:14.672089135 +0000 UTC"
Nov 02 13:43:29.338377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.319048    5034 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13327h46m45.353044747s"
Nov 02 13:43:29.338377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.319267    5034 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:43:29.338377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.320313    5034 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 13:43:29.338377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.321644    5034 log.go:25] "Validated CRI v1 runtime API"
Nov 02 13:43:29.338377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.323403    5034 log.go:25] "Validated CRI v1 image API"
Nov 02 13:43:29.338377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.324172    5034 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 13:43:29.338377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.326097    5034 fs.go:135] Filesystem UUIDs: map[5e50ca4f-00de-4ba8-ba44-cfa240489609:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9e0b04ea-c5a0-410d-809f-df9fefd195af:/dev/nvme0n1p4]
Nov 02 13:43:29.338630 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.326116    5034 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e5bc3a3ddad83a4131da521da3db8121936be915c02f2ef1ab89ac06c5df0f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e5bc3a3ddad83a4131da521da3db8121936be915c02f2ef1ab89ac06c5df0f8/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/177cfdfed6d877425545577f69af401ace0cb6d7f42a3976a12aca437670a2ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/177cfdfed6d877425545577f69af401ace0cb6d7f42a3976a12aca437670a2ef/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a181f38f91d1d321f8247113e0d9fd13a042aa0786d7a0eccaed1c82feb4130/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a181f38f91d1d321f8247113e0d9fd13a042aa0786d7a0eccaed1c82feb4130/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c95932f9108f4aed9cb7db3ab6750457f8867c4414da68250c7132e4419b6a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c95932f9108f4aed9cb7db3ab6750457f8867c4414da68250c7132e4419b6a5/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1caa65cb2d0470813f89c06842b8df0d9688ee5799a934234d23476c31c04867/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1caa65cb2d0470813f89c06842b8df0d9688ee5799a934234d23476c31c04867/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30179d4d8ff0b96d14fec5a86f48c3a5b04e6b73264111ecab3a796544789e4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30179d4d8ff0b96d14fec5a86f48c3a5b04e6b73264111ecab3a796544789e4c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30b87854575dd5d5baf607c9db02cd611b7f81465fe52a5b8ba15cbf5328b072/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30b87854575dd5d5baf607c9db02cd611b7f81465fe52a5b8ba15cbf5328b072/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3fd128a0f6e080fd4646150fee33128327829dd6ae32ca4797f76c8e35287cc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3fd128a0f6e080fd4646150fee33128327829dd6ae32ca4797f76c8e35287cc9/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c54c3e9ab3bcab4bbd3dc7ec62e43b33a3cfa789b1adb1cf482831ce5afa8c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c54c3e9ab3bcab4bbd3dc7ec62e43b33a3cfa789b1adb1cf482831ce5afa8c6/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/510c37485b484c9d4da49a3a5cbbd610d442bfe6f240ca2197019c0752c7b1e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/510c37485b484c9d4da49a3a5cbbd610d442bfe6f240ca2197019c0752c7b1e0/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5215d070599282003b08c100539b8ffd414603538e2834720bbc7b12979aeb9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5215d070599282003b08c100539b8ffd414603538e2834720bbc7b12979aeb9a/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e53990fbb91dd58e726bcba38adb28b827ed817e5b7357a968b5b7b6a1fab76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e53990fbb91dd58e726bcba38adb28b827ed817e5b7357a968b5b7b6a1fab76/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/645611df7ab78705ebae02c65af9ccd48c54ce0fe0200f88a4df8134bc435cf8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/645611df7ab78705ebae02c65af9ccd48c54ce0fe0200f88a4df8134bc435cf8/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69e32cae509e5d6868d134c1f4a1a1ba296d495157dbf4690787a29120c17b79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69e32cae509e5d6868d134c1f4a1a1ba296d495157dbf4690787a29120c17b79/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cd201012c26fed6351658b222ea0788f7f9813fcc30d7408e37e40c4b6fa864/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cd201012c26fed6351658b222ea0788f7f9813fcc30d7408e37e40c4b6fa864/userdata/shm major:0 minor:323 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6db9f5bd2d36b7fdca6a5b46f0deb14825a11f1e91be741e5d8db248f90a8706/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6db9f5bd2d36b7fdca6a5b46f0deb14825a11f1e91be741e5d8db248f90a8706/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81e9e260477d75bf602d68d90ce95eff15df2c45c8da9ce1069ab7c2b66778b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81e9e260477d75bf602d68d90ce95eff15df2c45c8da9ce1069ab7c2b66778b1/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cbfe03b6a690c4bc8c1215b4b789a1c6ab5bbbc9ddbae9b197d765c8b3cab8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cbfe03b6a690c4bc8c1215b4b789a1c6ab5bbbc9ddbae9b197d765c8b3cab8c/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93d9de324f0fadd867b9df3293b8f56a504c91e4f551b8db56b824d5da69fc65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93d9de324f0fadd867b9df3293b8f56a504c91e4f551b8db56b824d5da69fc65/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e1fa3d5c174e3f7eed0f34a4b335339ccfa8d91af01c486404663cbfac1247b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e1fa3d5c174e3f7eed0f34a4b335339ccfa8d91af01c486404663cbfac1247b/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b04e07673f0d267b355ade260ac8f3ff163299191886025e39e949e1d00a3e05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b04e07673f0d267b355ade260ac8f3ff163299191886025e39e949e1d00a3e05/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd1fe85b457b64df005e91ff653c06eb95b1a5477175808c82adb84758a4e403/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd1fe85b457b64df005e91ff653c06eb95b1a5477175808c82adb84758a4e403/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de8e7e42dd03bdf1530b5ac4d7bb83c122e4ff195ea9d67b197e983390925b42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de8e7e42dd03bdf1530b5ac4d7bb83c122e4ff195ea9d67b197e983390925b42/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df412506e405502b7f20b0e039f8c2e6cf9c9409107629ce7ad7a3c91aab1928/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df412506e405502b7f20b0e039f8c2e6cf9c9409107629ce7ad7a3c91aab1928/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f053214b2a15076cb42798094959229a8ce7676b8149a25e06089dd398798da5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f053214b2a15076cb42798094959229a8ce7676b8149a25e06089dd398798da5/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fc871305024afeb57f42699ecdccdfea0dd868ba79ebbcaed9d32ccc514caff8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fc871305024afeb57f42699ecdccdfea0dd868ba79ebbcaed9d32ccc514caff8/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00be7ba2-6f06-4ed5-a007-d0ac612177ee/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/00be7ba2-6f06-4ed5-a007-d0ac612177ee/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/028f7fde-fe8a-4d88-9c6c-54193c5c490b/volumes/kubernetes.io~projected/kube-api-access-hb92t:{mountpoint:/var/lib/kubelet/pods/028f7fde-fe8a-4d88-9c6c-54193c5c490b/volumes/kubernetes.io~projected/kube-api-access-hb92t major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fb4e264-b7b8-46de-b222-25331e6400c4/volumes/kubernetes.io~projected/kube-api-access-g4fv5:{mountpoint:/var/lib/kubelet/pods/0fb4e264-b7b8-46de-b222-25331e6400c4/volumes/kubernetes.io~projected/kube-api-access-g4fv5 major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fb4e264-b7b8-46de-b222-25331e6400c4/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/0fb4e264-b7b8-46de-b222-25331e6400c4/volumes/kubernetes.io~secret/signing-key major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/132bf469-fbc8-419c-b19a-17bcabfecce1/volumes/kubernetes.io~projected/kube-api-access-hsrph:{mountpoint:/var/lib/kubelet/pods/132bf469-fbc8-419c-b19a-17bcabfecce1/volumes/kubernetes.io~projected/kube-api-access-hsrph major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1859de72-db9b-4e60-ac47-b8360156ed1a/volumes/kubernetes.io~projected/kube-api-access-qfhqc:{mountpoint:/var/lib/kubelet/pods/1859de72-db9b-4e60-ac47-b8360156ed1a/volumes/kubernetes.io~projected/kube-api-access-qfhqc major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18d90fc8-23b1-4fca-a9e2-83eb8f4786d4/volumes/kubernetes.io~projected/kube-api-access-qdrlq:{mountpoint:/var/lib/kubelet/pods/18d90fc8-23b1-4fca-a9e2-83eb8f4786d4/volumes/kubernetes.io~projected/kube-api-access-qdrlq major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26f2db88-b241-451f-8f1e-9f8c68ad0e73/volumes/kubernetes.io~projected/kube-api-access-sjt9s:{mountpoint:/var/lib/kubelet/pods/26f2db88-b241-451f-8f1e-9f8c68ad0e73/volumes/kubernetes.io~projected/kube-api-access-sjt9s major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27e3e0a4-dc29-4e68-9344-f82ea07cb067/volumes/kubernetes.io~projected/kube-api-access-58czq:{mountpoint:/var/lib/kubelet/pods/27e3e0a4-dc29-4e68-9344-f82ea07cb067/volumes/kubernetes.io~projected/kube-api-access-58czq major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27e3e0a4-dc29-4e68-9344-f82ea07cb067/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/27e3e0a4-dc29-4e68-9344-f82ea07cb067/volumes/kubernetes.io~secret/serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ca56ffb-997f-4edb-9efa-0609042e8f08/volumes/kubernetes.io~projected/kube-api-access-8vw64:{mountpoint:/var/lib/kubelet/pods/2ca56ffb-997f-4edb-9efa-0609042e8f08/volumes/kubernetes.io~projected/kube-api-access-8vw64 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35f03797-afb7-4aa5-a841-c665e99feaa4/volumes/kubernetes.io~projected/kube-api-access-tzhlq:{mountpoint:/var/lib/kubelet/pods/35f03797-afb7-4aa5-a841-c665e99feaa4/volumes/kubernetes.io~projected/kube-api-access-tzhlq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/398a3f5c-7cbe-448d-8f16-3360c18a38bf/volumes/kubernetes.io~projected/kube-api-access-xxts2:{mountpoint:/var/lib/kubelet/pods/398a3f5c-7cbe-448d-8f16-3360c18a38bf/volumes/kubernetes.io~projected/kube-api-access-xxts2 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/398a3f5c-7cbe-448d-8f16-3360c18a38bf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/398a3f5c-7cbe-448d-8f16-3360c18a38bf/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b44af93-ab31-483d-9eab-3e996c90eee7/volumes/kubernetes.io~projected/kube-api-access-d4fcq:{mountpoint:/var/lib/kubelet/pods/3b44af93-ab31-483d-9eab-3e996c90eee7/volumes/kubernetes.io~projected/kube-api-access-d4fcq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bfd00d2-35ee-4067-8649-bdb54ee596a2/volumes/kubernetes.io~projected/kube-api-access-fzcbk:{mountpoint:/var/lib/kubelet/pods/3bfd00d2-35ee-4067-8649-bdb54ee596a2/volumes/kubernetes.io~projected/kube-api-access-fzcbk major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~projected/kube-api-access-q4fv4:{mountpoint:/var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~projected/kube-api-access-q4fv4 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4150923a-78f1-4f52-a0d5-271ec942bd1b/volumes/kubernetes.io~projected/kube-api-access-jk7vv:{mountpoint:/var/lib/kubelet/pods/4150923a-78f1-4f52-a0d5-271ec942bd1b/volumes/kubernetes.io~projected/kube-api-access-jk7vv major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4366155b-b272-4b13-b328-4d9f87325b84/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4366155b-b272-4b13-b328-4d9f87325b84/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4366155b-b272-4b13-b328-4d9f87325b84/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4366155b-b272-4b13-b328-4d9f87325b84/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4366155b-b272-4b13-b328-4d9f87325b84/volumes/kubernetes.io~projected/kube-api-access-98ktl:{mountpoint:/var/lib/kubelet/pods/4366155b-b272-4b13-b328-4d9f87325b84/volumes/kubernetes.io~projected/kube-api-access-98ktl major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/488bddc1-06e8-446c-b612-5ddeac1d8bdf/volumes/kubernetes.io~projected/kube-api-access-rs7rc:{mountpoint:/var/lib/kubelet/pods/488bddc1-06e8-446c-b612-5ddeac1d8bdf/volumes/kubernetes.io~projected/kube-api-access-rs7rc major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d26dd5a-118a-4d50-9661-22feeb24d158/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5d26dd5a-118a-4d50-9661-22feeb24d158/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~projected/kube-api-access-dbrcq:{mountpoint:/var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~projected/kube-api-access-dbrcq major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63881d4e-a3bf-4948-a369-c892baba23a8/volumes/kubernetes.io~projected/kube-api-access-f4p82:{mountpoint:/var/lib/kubelet/pods/63881d4e-a3bf-4948-a369-c892baba23a8/volumes/kubernetes.io~projected/kube-api-access-f4p82 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8646ac2d-cfd2-4b7c-929a-906815e6551f/volumes/kubernetes.io~projected/kube-api-access-dkpj5:{mountpoint:/var/lib/kubelet/pods/8646ac2d-cfd2-4b7c-929a-906815e6551f/volumes/kubernetes.io~projected/kube-api-access-dkpj5 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8646ac2d-cfd2-4b7c-929a-906815e6551f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8646ac2d-cfd2-4b7c-929a-906815e6551f/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8816a23c-15b8-4775-8853-6dc6cf7526c5/volumes/kubernetes.io~projected/kube-api-access-qjlx2:{mountpoint:/var/lib/kubelet/pods/8816a23c-15b8-4775-8853-6dc6cf7526c5/volumes/kubernetes.io~projected/kube-api-access-qjlx2 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91358478-4176-40c6-85b3-9e044fd2fa22/volumes/kubernetes.io~projected/kube-api-access-cc9f7:{mountpoint:/var/lib/kubelet/pods/91358478-4176-40c6-85b3-9e044fd2fa22/volumes/kubernetes.io~projected/kube-api-access-cc9f7 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91600010-0a54-4517-b9a7-1cc38b075abd/volumes/kubernetes.io~projected/kube-api-access-47rlg:{mountpoint:/var/lib/kubelet/pods/91600010-0a54-4517-b9a7-1cc38b075abd/volumes/kubernetes.io~projected/kube-api-access-47rlg major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99fc1fe5-048b-4196-b66c-771710cd73c0/volumes/kubernetes.io~projected/kube-api-access-thqql:{mountpoint:/var/lib/kubelet/pods/99fc1fe5-048b-4196-b66c-771710cd73c0/volumes/kubernetes.io~projected/kube-api-access-thqql major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2/volumes/kubernetes.io~projected/kube-api-access-4v5tx:{mountpoint:/var/lib/kubelet/pods/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2/volumes/kubernetes.io~projected/kube-api-access-4v5tx major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd/volumes/kubernetes.io~projected/kube-api-access-2bx75:{mountpoint:/var/lib/kubelet/pods/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd/volumes/kubernetes.io~projected/kube-api-access-2bx75 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d861fac3-737a-4cd7-abc6-4af75b4c1d67/volumes/kubernetes.io~projected/kube-api-access-5xskc:{mountpoint:/var/lib/kubelet/pods/d861fac3-737a-4cd7-abc6-4af75b4c1d67/volumes/kubernetes.io~projected/kube-api-access-5xskc major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d861fac3-737a-4cd7-abc6-4af75b4c1d67/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/d861fac3-737a-4cd7-abc6-4af75b4c1d67/volumes/kubernetes.io~secret/default-certificate major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d861fac3-737a-4cd7-abc6-4af75b4c1d67/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/d861fac3-737a-4cd7-abc6-4af75b4c1d67/volumes/kubernetes.io~secret/stats-auth major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df16fea1-6bca-47f3-b9bd-35716e3d50f3/volumes/kubernetes.io~projected/kube-api-access-v9zld:{mountpoint:/var/lib/kubelet/pods/df16fea1-6bca-47f3-b9bd-35716e3d50f3/volumes/kubernetes.io~projected/kube-api-access-v9zld major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df16fea1-6bca-47f3-b9bd-35716e3d50f3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/df16fea1-6bca-47f3-b9bd-35716e3d50f3/volumes/kubernetes.io~secret/cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea36ca35-7e60-41a3-b25a-0e39965f22bf/volumes/kubernetes.io~projected/kube-api-access-srhb7:{mountpoint:/var/lib/kubelet/pods/ea36ca35-7e60-41a3-b25a-0e39965f22bf/volumes/kubernetes.io~projected/kube-api-access-srhb7 major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46f80e0-332a-4c19-81a2-08cbb65f7fb8/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f46f80e0-332a-4c19-81a2-08cbb65f7fb8/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46f80e0-332a-4c19-81a2-08cbb65f7fb8/volumes/kubernetes.io~projected/kube-api-access-n8dr7:{mountpoint:/var/lib/kubelet/pods/f46f80e0-332a-4c19-81a2-08cbb65f7fb8/volumes/kubernetes.io~projected/kube-api-access-n8dr7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f46f80e0-332a-4c19-81a2-08cbb65f7fb8/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f46f80e0-332a-4c19-81a2-08cbb65f7fb8/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb318d8f-e7de-4185-96cd-d592b304248a/volumes/kubernetes.io~projected/kube-api-access-47vrz:{mountpoint:/var/lib/kubelet/pods/fb318d8f-e7de-4185-96cd-d592b304248a/volumes/kubernetes.io~projected/kube-api-access-47vrz major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb81289b-9a76-4529-a6f5-649ea7c85d67/volumes/kubernetes.io~projected/kube-api-access-vlhvr:{mountpoint:/var/lib/kubelet/pods/fb81289b-9a76-4529-a6f5-649ea7c85d67/volumes/kubernetes.io~projected/kube-api-access-vlhvr major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb81289b-9a76-4529-a6f5-649ea7c85d67/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/fb81289b-9a76-4529-a6f5-649ea7c85d67/volumes/kubernetes.io~secret/serving-cert major:0 minor:226 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/8eda9726952f550496096a7957b15c607e65ca7ac8d7725b067d7c90ab545370/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d5132e2f6f6ac872ffa5aa81e50b563df1d926ea56857796e3cf198a7fdc6c52/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6090715e0e49532fcbfe8b15943d0cd8de6a194213fb172863c0685d24c10c36/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3b9e5972747d56a077e72b206e7edfcefd4b4f775acd49cab95a72d22406757f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f2ec69198e5d50dc781b22ebd08308fcd9995a1873ccc9960833a8913d4feb0c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8c17dcdd097f810957a010c5d56b29a304173554c1ff288969d250b679827684/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/0ef683c779b7f38607964ab4a679c689750c66c136f85c6d8db57679468e7a7a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3f55896144741abd77bb60c6d65a2e10b9284474ba68f15cf21838d08b5c19a8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/0ae2f78eb40501ec334a5692f225a71d7cefb899f834fcdd00c76633513bdace/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/2000b49bf07ea814d99e41735af0b47ded4e8fe9a36898465e31970b1eceb5c6/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/65694eaa690692246c60b221956a3fce656c88d100b8ff594e65bdcb11a45fd6/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/037a7f169b0452050a01a36214ff41c71a0e43abff971821ef439e5f91ace88f/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/37d34e3ec7da5d9675e79a4fe8d108fae1fab1b81bc341a91ba990b3028cd373/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/61dbde47f3b0d06dfcbf2cb185ef6e5943989e5ed9ccfc8fad746f178ab8b7e5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2350a9695b337d41fb3522743aab4c05bd9eb55cc3298d5359f2cf052cd3c0d2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/7f620438560fec8b62cb8012426ccf39bff5c2e8f1ec9b83f122ba9ccf3a4b8d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/089c580235eabd81a606e4f46a1e5930f74a28163962327547a7841e50a048e1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a573d2222db4eba4104923ec4137fb781af1248c066a7ec1b36c0b58fb385fe8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/cb6385299cc2646d7369b7c755af1500a6616fbe97ea3a064f3e4983cfe04159/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/59609bd07b17f7303253a6950f61199814bcee6d7949b45f3500d4abe5bf529e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/84d7c4071d4802101ef4d92dffdc83feef29a81d27d20e115ee181b5076b811a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/fcddd76126fd582ffc28862a6df285841be5e2924a64068f7bcd4025412235a8/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/8944e8746b612fb61fe7fd62911cccd47741bc0070fd2a5fe414da23bc896ae2/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/88b082ef012f52e430e2aacd28e61e98086a2a42751f7f47e25e0e4c40d3e229/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/1ff1ab7f2407b86b5e3de6401763d00dd87fdbd9b7f3055a1913c98d1b6dfaa3/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/d1b66900a185342f51f3de92ab1830ba33d5a8ce41a5676c85464701a6fb4686/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d803ff370de92e968d752e0da42d6cfb69e53ae4052a10af1972b366db6590d5/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/4af3fe02fcdfda01d55d62d991b11cb4cb37a533448e70e0907d8e4e07255a3c/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/2a2d7677be3437375146ea02468e70b716f53c6559f770796dfebc8ce78f1ca2/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ded5ce7f66be9e699e8d1968005d6ddac91f06d89616d8ef24b222a7a0abb971/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/0a07eb113d2b2a267620f6f582c13ccbf6e770cb3e9dc07f6964ccb34a8d0467/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/b13f94838f10aab644609c11b80ea164ae7bead13471d7d2f959071c407d43ba/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/8f6d9f2c0010b8eef975c639a105b58623d9664b1e09bb06f6f522cb801efc46/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/c4b18b18f1e4da1254ed8fc8098e7f0702559aedd1c9e785a6b466e9ae299dbc/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/cd1e70a1758f14ebb721334617430e7527aaa83b8f26993dcfaeffe1f16c5eb3/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/bb3a87ecc6ac64ffe44a04c18a8006018cabd44bfca4b589eb35a22c5a570a8c/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/de1cf42ef435bbe6dbcb82cc636d52e7d87c18fb6ece8cea486a5af621ac7535/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/c97e556bbe48e99d91851980f4135a4ab205dd60e543da6674efd9e6ed1fc3cb/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/0d23c162dc2cb97ffd46da01f1955c4ec594b3f1a017d25486b3d3137ef238f6/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/3a294574a1c1b83544ae7c7ea2c351333bf42aa8eeb7b77bffb6c3dca0935c90/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/9e596655321671411f28ec6054319fe3f8af973d4e94b4a9cf1286d8d3b9a022/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/fafdd056a60644504f772e3e364263d5dcac4166ff5c07b64485f02dfecbfb40/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/23660fa74c85f4c1d12b96fdfe5242b6e95b299b48d479748a3176384beaafc8/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/7dbe9c0a88605eaac4e3334b5af8c90d7fafb0a7ae6d18f72642b9dad678be8f/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/c5934ccbcde6f760497ec8e358ad21b4d320083b5c7592c7d70f3f2d3a1dcd13/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/c6b1b130a033fd2800081d709f56f7eddfa3e340e76b782bf24b9889b7106d15/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/4e1a997d0feaf41ce3ec662cbcb65d7f5b8f5e682ce2ea79ab98172ba036eb47/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/afb20eb459281865bb15d9bf1a3e8da11663ca1ff959f80689d159110635691b/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/5ee7ed2d9dc12d0b9762faf6dbe63579577d1ff7578dbe54f31c96a0b6519e17/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/73a56f5baa8e671563c8ff5f295bf3eff96622ab10ada1a77db663afa8c16373/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/e97e508d6ac9ff9fdacb3ebc3cae0344baf782307e93d7117df7a1e6bd074d61/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/85f1cdd875a5a86e09cc72e9aa2f1078803ce16380d943dd79185a5134b0874e/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/4fc73e7e9a307f0f8be129cd81c2f5979885bcc0b35004bb06d0f611bd1cbff3/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ef4ac7ded8c55ba4ebeb79c2297d33608b784b49623189a40ed0faa67a52e71c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5bce3d0b0ea956a90bde161b90ee310d205dad689aa18da858b95a430509a1dd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e110ddacc159a02be78019a20c11ad275c2eb608d74cd80b4d89743bf2659719/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b0f0cd7e711f7373274fd08c3e0caf055d6c92abfbb0f97388450616b9248944/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/01aceb2871dd89e40b361ef0c2d9606b5a0c205764bcb4c2ca844a31a41182f9/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/41616b708ef046dbb486aa59e295c7e0d1cdf41d9d2b6b33fd59db946f5a78b9/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/2b654c07de70a4a524db9e9c9e114c1f89392f8d05011a7b2c1ba7e221749f15/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3e5c32478960df4a315e89f873df110d31512414da10be58a8a4f8eb53b3d30c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/6b40cdaedb41336ac059fdf3583d5d88694eaa3e9923d60b4f455abf84bac0d5/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.337149    5034 manager.go:217] Machine: {Timestamp:2026-11-02 13:43:29.33634023 +0000 UTC m=+0.087644659 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094023 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec238c3f97d96f6a70d8ccbed1707f65 SystemUUID:ec238c3f-97d9-6f6a-70d8-ccbed1707f65 BootID:56f3a6b2-2d88-4534-bce7-ba50c0584984 Filesystems:[{Device:/var/lib/kubelet/pods/0fb4e264-b7b8-46de-b222-25331e6400c4/volumes/kubernetes.io~projected/kube-api-access-g4fv5 DeviceMajor:0 DeviceMinor:349 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c54c3e9ab3bcab4bbd3dc7ec62e43b33a3cfa789b1adb1cf482831ce5afa8c6/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35f03797-afb7-4aa5-a841-c665e99feaa4/volumes/kubernetes.io~projected/kube-api-access-tzhlq DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69e32cae509e5d6868d134c1f4a1a1ba296d495157dbf4690787a29120c17b79/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd/volumes/kubernetes.io~projected/kube-api-access-2bx75 DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46f80e0-332a-4c19-81a2-08cbb65f7fb8/volumes/kubernetes.io~projected/kube-api-access-n8dr7 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bfd00d2-35ee-4067-8649-bdb54ee596a2/volumes/kubernetes.io~projected/kube-api-access-fzcbk DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/27e3e0a4-dc29-4e68-9344-f82ea07cb067/volumes/kubernetes.io~projected/kube-api-access-58czq DeviceMajor:0 DeviceMinor:235 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5215d070599282003b08c100539b8ffd414603538e2834720bbc7b12979aeb9a/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/132bf469-fbc8-419c-b19a-17bcabfecce1/volumes/kubernetes.io~projected/kube-api-access-hsrph DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46f80e0-332a-4c19-81a2-08cbb65f7fb8/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a181f38f91d1d321f8247113e0d9fd13a042aa0786d7a0eccaed1c82feb4130/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1caa65cb2d0470813f89c06842b8df0d9688ee5799a934234d23476c31c04867/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30b87854575dd5d5baf607c9db02cd611b7f81465fe52a5b8ba15cbf5328b072/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cd201012c26fed6351658b222ea0788f7f9813fcc30d7408e37e40c4b6fa864/userdata/shm DeviceMajor:0 DeviceMinor:323 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/510c37485b484c9d4da49a3a5cbbd610d442bfe6f240ca2197019c0752c7b1e0/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cbfe03b6a690c4bc8c1215b4b789a1c6ab5bbbc9ddbae9b197d765c8b3cab8c/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91358478-4176-40c6-85b3-9e044fd2fa22/volumes/kubernetes.io~projected/kube-api-access-cc9f7 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/91600010-0a54-4517-b9a7-1cc38b075abd/volumes/kubernetes.io~projected/kube-api-access-47rlg DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fc871305024afeb57f42699ecdccdfea0dd868ba79ebbcaed9d32ccc514caff8/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d861fac3-737a-4cd7-abc6-4af75b4c1d67/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/488bddc1-06e8-446c-b612-5ddeac1d8bdf/volumes/kubernetes.io~projected/kube-api-access-rs7rc DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/00be7ba2-6f06-4ed5-a007-d0ac612177ee/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3fd128a0f6e080fd4646150fee33128327829dd6ae32ca4797f76c8e35287cc9/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/028f7fde-fe8a-4d88-9c6c-54193c5c490b/volumes/kubernetes.io~projected/kube-api-access-hb92t DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c95932f9108f4aed9cb7db3ab6750457f8867c4414da68250c7132e4419b6a5/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4366155b-b272-4b13-b328-4d9f87325b84/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/398a3f5c-7cbe-448d-8f16-3360c18a38bf/volumes/kubernetes.io~projected/kube-api-access-xxts2 DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6db9f5bd2d36b7fdca6a5b46f0deb14825a11f1e91be741e5d8db248f90a8706/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~projected/kube-api-access-q4fv4 DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/177cfdfed6d877425545577f69af401ace0cb6d7f42a3976a12aca437670a2ef/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/63881d4e-a3bf-4948-a369-c892baba23a8/volumes/kubernetes.io~projected/kube-api-access-f4p82 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/645611df7ab78705ebae02c65af9ccd48c54ce0fe0200f88a4df8134bc435cf8/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1859de72-db9b-4e60-ac47-b8360156ed1a/volumes/kubernetes.io~projected/kube-api-access-qfhqc DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4150923a-78f1-4f52-a0d5-271ec942bd1b/volumes/kubernetes.io~projected/kube-api-access-jk7vv DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2/volumes/kubernetes.io~projected/kube-api-access-4v5tx DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/398a3f5c-7cbe-448d-8f16-3360c18a38bf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f46f80e0-332a-4c19-81a2-08cbb65f7fb8/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b04e07673f0d267b355ade260ac8f3ff163299191886025e39e949e1d00a3e05/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e1fa3d5c174e3f7eed0f34a4b335339ccfa8d91af01c486404663cbfac1247b/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd1fe85b457b64df005e91ff653c06eb95b1a5477175808c82adb84758a4e403/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81e9e260477d75bf602d68d90ce95eff15df2c45c8da9ce1069ab7c2b66778b1/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f053214b2a15076cb42798094959229a8ce7676b8149a25e06089dd398798da5/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8646ac2d-cfd2-4b7c-929a-906815e6551f/volumes/kubernetes.io~projected/kube-api-access-dkpj5 DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fb4e264-b7b8-46de-b222-25331e6400c4/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:348 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30179d4d8ff0b96d14fec5a86f48c3a5b04e6b73264111ecab3a796544789e4c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d861fac3-737a-4cd7-abc6-4af75b4c1d67/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/27e3e0a4-dc29-4e68-9344-f82ea07cb067/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb81289b-9a76-4529-a6f5-649ea7c85d67/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df412506e405502b7f20b0e039f8c2e6cf9c9409107629ce7ad7a3c91aab1928/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df16fea1-6bca-47f3-b9bd-35716e3d50f3/volumes/kubernetes.io~projected/kube-api-access-v9zld DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26f2db88-b241-451f-8f1e-9f8c68ad0e73/volumes/kubernetes.io~projected/kube-api-access-sjt9s DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de8e7e42dd03bdf1530b5ac4d7bb83c122e4ff195ea9d67b197e983390925b42/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b44af93-ab31-483d-9eab-3e996c90eee7/volumes/kubernetes.io~projected/kube-api-access-d4fcq DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8646ac2d-cfd2-4b7c-929a-906815e6551f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fb318d8f-e7de-4185-96cd-d592b304248a/volumes/kubernetes.io~projected/kube-api-access-47vrz DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea36ca35-7e60-41a3-b25a-0e39965f22bf/volumes/kubernetes.io~projected/kube-api-access-srhb7 DeviceMajor:0 DeviceMinor:331 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99fc1fe5-048b-4196-b66c-771710cd73c0/volumes/kubernetes.io~projected/kube-api-access-thqql DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/18d90fc8-23b1-4fca-a9e2-83eb8f4786d4/volumes/kubernetes.io~projected/kube-api-access-qdrlq DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d861fac3-737a-4cd7-abc6-4af75b4c1d67/volumes/kubernetes.io~projected/kube-api-access-5xskc DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93d9de324f0fadd867b9df3293b8f56a504c91e4f551b8db56b824d5da69fc65/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e5bc3a3ddad83a4131da521da3db8121936be915c02f2ef1ab89ac06c5df0f8/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4366155b-b272-4b13-b328-4d9f87325b84/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8816a23c-15b8-4775-8853-6dc6cf7526c5/volumes/kubernetes.io~projected/kube-api-access-qjlx2 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5d26dd5a-118a-4d50-9661-22feeb24d158/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e53990fbb91dd58e726bcba38adb28b827ed817e5b7357a968b5b7b6a1fab76/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df16fea1-6bca-47f3-b9bd-35716e3d50f3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ca56ffb-997f-4edb-9efa-0609042e8f08/volumes/kubernetes.io~projected/kube-api-access-8vw64 DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~projected/kube-api-access-dbrcq DeviceMajor:0 DeviceMinor:234 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4366155b-b272-4b13-b328-4d9f87325b84/volumes/kubernetes.io~projected/kube-api-access-98ktl DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:236 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb81289b-9a76-4529-a6f5-649ea7c85d67/volumes/kubernetes.io~projected/kube-api-access-vlhvr DeviceMajor:0 DeviceMinor:237 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0e5bc3a3ddad83a MacAddress:2e:1d:ea:37:ea:e3 Speed:10000 Mtu:8901} {Name:1a181f38f91d1d3 MacAddress:ba:75:72:b0:6d:bf Speed:10000 Mtu:8901} {Name:1c95932f9108f4a MacAddress:06:48:c7:df:3f:06 Speed:10000 Mtu:8901} {Name:1caa65cb2d04708 MacAddress:3a:c8:96:f4:77:ee Speed:10000 Mtu:8901} {Name:3fd128a0f6e080f MacAddress:42:97:28:b5:c2:8a Speed:10000 Mtu:8901} {Name:5215d0705992820 MacAddress:fa:15:2e:fe:2a:bb Speed:10000 Mtu:8901} {Name:69e32cae509e5d6 MacAddress:66:86:82:46:cb:3a Speed:10000 Mtu:8901} {Name:6cd201012c26fed MacAddress:96:fe:d3:7c:6d:7c Speed:10000 Mtu:8901} {Name:6db9f5bd2d36b7f MacAddress:5a:ba:0b:45:5e:4b Speed:10000 Mtu:8901} {Name:8cbfe03b6a690c4 MacAddress:8a:55:fe:1b:54:2c Speed:10000 Mtu:8901} {Name:93d9de324f0fadd MacAddress:fe:1d:78:84:1b:c6 Speed:10000 Mtu:8901} {Name:9e1fa3d5c174e3f MacAddress:c2:d6:a1:0c:02:9a Speed:10000 Mtu:8901} {Name:b04e07673f0d267 MacAddress:3a:39:aa:26:fa:e1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f3:5b:7b:bd Speed:0 Mtu:9001} {Name:br-int MacAddress:1e:92:a9:8b:8b:d3 Speed:0 Mtu:8901} {Name:de8e7e42dd03bdf MacAddress:c6:e3:a5:ac:0f:06 Speed:10000 Mtu:8901} {Name:df412506e405502 MacAddress:42:e8:d8:64:ea:ce Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f3:5b:7b:bd Speed:0 Mtu:9001} {Name:f053214b2a15076 MacAddress:8a:5b:31:11:95:52 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:96:59:eb:77:63:8b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:0c:fd:30:c4:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.337811    5034 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.337869    5034 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338136    5034 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338156    5034 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-26.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338272    5034 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338280    5034 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338290    5034 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338313    5034 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338332    5034 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338498    5034 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338596    5034 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338627    5034 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338639    5034 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338649    5034 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338657    5034 kubelet.go:405] "Adding apiserver pod source"
Nov 02 13:43:29.339011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.338664    5034 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 13:43:29.340055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.339359    5034 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 13:43:29.340055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.339369    5034 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:43:29.340055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.339856    5034 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 13:43:29.341997 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.341981    5034 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 13:43:29.341997 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.341993    5034 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 13:43:29.342100 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342080    5034 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 13:43:29.342257 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342245    5034 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 13:43:29.342287 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342275    5034 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 13:43:29.342367 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342358    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 13:43:29.342411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342372    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 13:43:29.342411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342379    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 13:43:29.342411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342384    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 13:43:29.342411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342390    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 13:43:29.342411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342395    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 13:43:29.342411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342401    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 13:43:29.342411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342406    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 13:43:29.342411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342413    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 13:43:29.342675 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342419    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 13:43:29.342675 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342427    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 13:43:29.342675 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342436    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 13:43:29.342675 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342462    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 13:43:29.342675 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.342468    5034 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 13:43:29.346095 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.346081    5034 server.go:1263] "Started kubelet"
Nov 02 13:43:29.346335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.346274    5034 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 13:43:29.346598 ip-10-0-139-26 systemd[1]: Started Kubernetes Kubelet.
Nov 02 13:43:29.351429 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.351357    5034 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 13:43:29.351530 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.351459    5034 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 13:43:29.352111 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.352082    5034 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 13:43:29.353617 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.353597    5034 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 13:43:29.356889 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.356870    5034 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 13:43:29.356989 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.356890    5034 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 13:43:29.356989 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.356964    5034 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 13:37:50 +0000 UTC" deadline="2028-06-19 05:56:51.488577242 +0000 UTC"
Nov 02 13:43:29.357091 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.356984    5034 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14272h13m22.1315962s"
Nov 02 13:43:29.357247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.357232    5034 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 13:43:29.357247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.357248    5034 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 13:43:29.357364 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.357341    5034 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 13:43:29.359832 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:29.357049    5034 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-26.ec2.internal\" not found"
Nov 02 13:43:29.360084 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360031    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="132bf469-fbc8-419c-b19a-17bcabfecce1" volumeName="kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-binary-copy" seLinuxMountContext=""
Nov 02 13:43:29.360122 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360093    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4366155b-b272-4b13-b328-4d9f87325b84" volumeName="kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-etc-tuned" seLinuxMountContext=""
Nov 02 13:43:29.360122 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360107    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99fc1fe5-048b-4196-b66c-771710cd73c0" volumeName="kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle" seLinuxMountContext=""
Nov 02 13:43:29.360191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360119    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d861fac3-737a-4cd7-abc6-4af75b4c1d67" volumeName="kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-default-certificate" seLinuxMountContext=""
Nov 02 13:43:29.360191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360132    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb81289b-9a76-4529-a6f5-649ea7c85d67" volumeName="kubernetes.io/projected/fb81289b-9a76-4529-a6f5-649ea7c85d67-kube-api-access-vlhvr" seLinuxMountContext=""
Nov 02 13:43:29.360191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360143    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="028f7fde-fe8a-4d88-9c6c-54193c5c490b" volumeName="kubernetes.io/empty-dir/028f7fde-fe8a-4d88-9c6c-54193c5c490b-data-volume" seLinuxMountContext=""
Nov 02 13:43:29.360191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360153    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa80f89-f24d-4f86-940c-864dbf177a5c" volumeName="kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 13:43:29.360191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360164    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa80f89-f24d-4f86-940c-864dbf177a5c" volumeName="kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates" seLinuxMountContext=""
Nov 02 13:43:29.360191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360177    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca56ffb-997f-4edb-9efa-0609042e8f08" volumeName="kubernetes.io/configmap/2ca56ffb-997f-4edb-9efa-0609042e8f08-telemetry-config" seLinuxMountContext=""
Nov 02 13:43:29.360191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360188    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa80f89-f24d-4f86-940c-864dbf177a5c" volumeName="kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360199    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="633c4858-9dc5-4049-aec3-8a14171652ae" volumeName="kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360210    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63881d4e-a3bf-4948-a369-c892baba23a8" volumeName="kubernetes.io/configmap/63881d4e-a3bf-4948-a369-c892baba23a8-config-volume" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360222    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91358478-4176-40c6-85b3-9e044fd2fa22" volumeName="kubernetes.io/projected/91358478-4176-40c6-85b3-9e044fd2fa22-kube-api-access-cc9f7" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360233    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46f80e0-332a-4c19-81a2-08cbb65f7fb8" volumeName="kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-config" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360255    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e3e0a4-dc29-4e68-9344-f82ea07cb067" volumeName="kubernetes.io/configmap/27e3e0a4-dc29-4e68-9344-f82ea07cb067-config" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360266    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35f03797-afb7-4aa5-a841-c665e99feaa4" volumeName="kubernetes.io/projected/35f03797-afb7-4aa5-a841-c665e99feaa4-kube-api-access-tzhlq" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360278    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa80f89-f24d-4f86-940c-864dbf177a5c" volumeName="kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360290    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4366155b-b272-4b13-b328-4d9f87325b84" volumeName="kubernetes.io/projected/4366155b-b272-4b13-b328-4d9f87325b84-kube-api-access-98ktl" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360301    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8646ac2d-cfd2-4b7c-929a-906815e6551f" volumeName="kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360313    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99fc1fe5-048b-4196-b66c-771710cd73c0" volumeName="kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360324    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca56ffb-997f-4edb-9efa-0609042e8f08" volumeName="kubernetes.io/projected/2ca56ffb-997f-4edb-9efa-0609042e8f08-kube-api-access-8vw64" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360336    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8646ac2d-cfd2-4b7c-929a-906815e6551f" volumeName="kubernetes.io/secret/8646ac2d-cfd2-4b7c-929a-906815e6551f-serving-cert" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360348    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b34b2c6f-210f-4256-8383-16f6a3a360bb" volumeName="kubernetes.io/configmap/b34b2c6f-210f-4256-8383-16f6a3a360bb-nginx-conf" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360360    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd" volumeName="kubernetes.io/projected/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-kube-api-access-2bx75" seLinuxMountContext=""
Nov 02 13:43:29.360370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360372    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df16fea1-6bca-47f3-b9bd-35716e3d50f3" volumeName="kubernetes.io/secret/df16fea1-6bca-47f3-b9bd-35716e3d50f3-cert" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360383    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46f80e0-332a-4c19-81a2-08cbb65f7fb8" volumeName="kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360396    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb4e264-b7b8-46de-b222-25331e6400c4" volumeName="kubernetes.io/secret/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-key" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360414    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398a3f5c-7cbe-448d-8f16-3360c18a38bf" volumeName="kubernetes.io/projected/398a3f5c-7cbe-448d-8f16-3360c18a38bf-kube-api-access-xxts2" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360431    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63881d4e-a3bf-4948-a369-c892baba23a8" volumeName="kubernetes.io/projected/63881d4e-a3bf-4948-a369-c892baba23a8-kube-api-access-f4p82" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360442    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea36ca35-7e60-41a3-b25a-0e39965f22bf" volumeName="kubernetes.io/projected/ea36ca35-7e60-41a3-b25a-0e39965f22bf-kube-api-access-srhb7" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360459    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46f80e0-332a-4c19-81a2-08cbb65f7fb8" volumeName="kubernetes.io/projected/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-kube-api-access-n8dr7" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360470    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="132bf469-fbc8-419c-b19a-17bcabfecce1" volumeName="kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360483    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398a3f5c-7cbe-448d-8f16-3360c18a38bf" volumeName="kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-config" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360495    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="633c4858-9dc5-4049-aec3-8a14171652ae" volumeName="kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360508    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="633c4858-9dc5-4049-aec3-8a14171652ae" volumeName="kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360519    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63881d4e-a3bf-4948-a369-c892baba23a8" volumeName="kubernetes.io/empty-dir/63881d4e-a3bf-4948-a369-c892baba23a8-tmp-dir" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360529    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b44af93-ab31-483d-9eab-3e996c90eee7" volumeName="kubernetes.io/projected/3b44af93-ab31-483d-9eab-3e996c90eee7-kube-api-access-d4fcq" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360541    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d26dd5a-118a-4d50-9661-22feeb24d158" volumeName="kubernetes.io/configmap/5d26dd5a-118a-4d50-9661-22feeb24d158-konnectivity-ca" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360552    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99fc1fe5-048b-4196-b66c-771710cd73c0" volumeName="kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360564    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bfd00d2-35ee-4067-8649-bdb54ee596a2" volumeName="kubernetes.io/configmap/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-config" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360580    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4366155b-b272-4b13-b328-4d9f87325b84" volumeName="kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-tmp" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360590    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99fc1fe5-048b-4196-b66c-771710cd73c0" volumeName="kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360601    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2d67e5a-851c-4fdd-befa-c0a97e93d5d2" volumeName="kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-daemon-config" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360612    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df16fea1-6bca-47f3-b9bd-35716e3d50f3" volumeName="kubernetes.io/projected/df16fea1-6bca-47f3-b9bd-35716e3d50f3-kube-api-access-v9zld" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360623    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e3e0a4-dc29-4e68-9344-f82ea07cb067" volumeName="kubernetes.io/secret/27e3e0a4-dc29-4e68-9344-f82ea07cb067-serving-cert" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360634    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8646ac2d-cfd2-4b7c-929a-906815e6551f" volumeName="kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-snapshots" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360646    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2d67e5a-851c-4fdd-befa-c0a97e93d5d2" volumeName="kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cni-binary-copy" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360658    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2d67e5a-851c-4fdd-befa-c0a97e93d5d2" volumeName="kubernetes.io/projected/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-kube-api-access-4v5tx" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360668    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46f80e0-332a-4c19-81a2-08cbb65f7fb8" volumeName="kubernetes.io/secret/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360680    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bfd00d2-35ee-4067-8649-bdb54ee596a2" volumeName="kubernetes.io/projected/3bfd00d2-35ee-4067-8649-bdb54ee596a2-kube-api-access-fzcbk" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360691    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="132bf469-fbc8-419c-b19a-17bcabfecce1" volumeName="kubernetes.io/projected/132bf469-fbc8-419c-b19a-17bcabfecce1-kube-api-access-hsrph" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360702    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18d90fc8-23b1-4fca-a9e2-83eb8f4786d4" volumeName="kubernetes.io/projected/18d90fc8-23b1-4fca-a9e2-83eb8f4786d4-kube-api-access-qdrlq" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360714    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26f2db88-b241-451f-8f1e-9f8c68ad0e73" volumeName="kubernetes.io/empty-dir/26f2db88-b241-451f-8f1e-9f8c68ad0e73-tmp-dir" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360731    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb4e264-b7b8-46de-b222-25331e6400c4" volumeName="kubernetes.io/projected/0fb4e264-b7b8-46de-b222-25331e6400c4-kube-api-access-g4fv5" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360742    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa80f89-f24d-4f86-940c-864dbf177a5c" volumeName="kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca" seLinuxMountContext=""
Nov 02 13:43:29.360744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360754    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa80f89-f24d-4f86-940c-864dbf177a5c" volumeName="kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360794    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488bddc1-06e8-446c-b612-5ddeac1d8bdf" volumeName="kubernetes.io/projected/488bddc1-06e8-446c-b612-5ddeac1d8bdf-kube-api-access-rs7rc" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360807    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="633c4858-9dc5-4049-aec3-8a14171652ae" volumeName="kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360819    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="633c4858-9dc5-4049-aec3-8a14171652ae" volumeName="kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360831    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8816a23c-15b8-4775-8853-6dc6cf7526c5" volumeName="kubernetes.io/projected/8816a23c-15b8-4775-8853-6dc6cf7526c5-kube-api-access-qjlx2" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360842    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91600010-0a54-4517-b9a7-1cc38b075abd" volumeName="kubernetes.io/projected/91600010-0a54-4517-b9a7-1cc38b075abd-kube-api-access-47rlg" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360854    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="132bf469-fbc8-419c-b19a-17bcabfecce1" volumeName="kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360866    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398a3f5c-7cbe-448d-8f16-3360c18a38bf" volumeName="kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-trusted-ca" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360877    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fa80f89-f24d-4f86-940c-864dbf177a5c" volumeName="kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360888    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd" volumeName="kubernetes.io/configmap/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-serviceca" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360900    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f46f80e0-332a-4c19-81a2-08cbb65f7fb8" volumeName="kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-env-overrides" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360911    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26f2db88-b241-451f-8f1e-9f8c68ad0e73" volumeName="kubernetes.io/projected/26f2db88-b241-451f-8f1e-9f8c68ad0e73-kube-api-access-sjt9s" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360924    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4150923a-78f1-4f52-a0d5-271ec942bd1b" volumeName="kubernetes.io/projected/4150923a-78f1-4f52-a0d5-271ec942bd1b-kube-api-access-jk7vv" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360934    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="633c4858-9dc5-4049-aec3-8a14171652ae" volumeName="kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360945    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8646ac2d-cfd2-4b7c-929a-906815e6551f" volumeName="kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-service-ca-bundle" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360957    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8646ac2d-cfd2-4b7c-929a-906815e6551f" volumeName="kubernetes.io/projected/8646ac2d-cfd2-4b7c-929a-906815e6551f-kube-api-access-dkpj5" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360968    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d861fac3-737a-4cd7-abc6-4af75b4c1d67" volumeName="kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-stats-auth" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360979    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb318d8f-e7de-4185-96cd-d592b304248a" volumeName="kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.360990    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb81289b-9a76-4529-a6f5-649ea7c85d67" volumeName="kubernetes.io/configmap/fb81289b-9a76-4529-a6f5-649ea7c85d67-config" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361003    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00be7ba2-6f06-4ed5-a007-d0ac612177ee" volumeName="kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361015    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27e3e0a4-dc29-4e68-9344-f82ea07cb067" volumeName="kubernetes.io/projected/27e3e0a4-dc29-4e68-9344-f82ea07cb067-kube-api-access-58czq" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361029    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398a3f5c-7cbe-448d-8f16-3360c18a38bf" volumeName="kubernetes.io/secret/398a3f5c-7cbe-448d-8f16-3360c18a38bf-serving-cert" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361040    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d26dd5a-118a-4d50-9661-22feeb24d158" volumeName="kubernetes.io/secret/5d26dd5a-118a-4d50-9661-22feeb24d158-agent-certs" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361051    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="633c4858-9dc5-4049-aec3-8a14171652ae" volumeName="kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361061    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8646ac2d-cfd2-4b7c-929a-906815e6551f" volumeName="kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-tmp" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361072    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91358478-4176-40c6-85b3-9e044fd2fa22" volumeName="kubernetes.io/configmap/91358478-4176-40c6-85b3-9e044fd2fa22-iptables-alerter-script" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361085    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d861fac3-737a-4cd7-abc6-4af75b4c1d67" volumeName="kubernetes.io/projected/d861fac3-737a-4cd7-abc6-4af75b4c1d67-kube-api-access-5xskc" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361097    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="028f7fde-fe8a-4d88-9c6c-54193c5c490b" volumeName="kubernetes.io/projected/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-api-access-hb92t" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361109    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fb4e264-b7b8-46de-b222-25331e6400c4" volumeName="kubernetes.io/configmap/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-cabundle" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361123    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1859de72-db9b-4e60-ac47-b8360156ed1a" volumeName="kubernetes.io/projected/1859de72-db9b-4e60-ac47-b8360156ed1a-kube-api-access-qfhqc" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361135    5034 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb81289b-9a76-4529-a6f5-649ea7c85d67" volumeName="kubernetes.io/secret/fb81289b-9a76-4529-a6f5-649ea7c85d67-serving-cert" seLinuxMountContext=""
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361145    5034 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361152    5034 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361644    5034 factory.go:55] Registering systemd factory
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361656    5034 factory.go:223] Registration of the systemd container factory successfully
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361893    5034 factory.go:153] Registering CRI-O factory
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361903    5034 factory.go:223] Registration of the crio container factory successfully
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361949    5034 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361962    5034 factory.go:103] Registering Raw factory
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.361973    5034 manager.go:1199] Started watching for new ooms in manager
Nov 02 13:43:29.363032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.362227    5034 manager.go:319] Starting recovery of all containers
Nov 02 13:43:29.378302 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.378279    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/1.log"
Nov 02 13:43:29.380042 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.380013    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/1.log"
Nov 02 13:43:29.384584 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.384551    5034 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 13:43:29.386133 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.386110    5034 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 13:43:29.386133 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.386131    5034 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 13:43:29.386277 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.386151    5034 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 13:43:29.386277 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:29.386194    5034 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 13:43:29.401344 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.401326    5034 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 13:43:29.401988 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.401968    5034 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:29.415904 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.415847    5034 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:29.416050 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.416031    5034 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:29.416206 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.416190    5034 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 13:43:29.422160 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.422127    5034 generic.go:356] "Generic (PLEG): container finished" podID="7b3a0abeec74791fe4e4176d1d343021" containerID="d1afcc17a1ae9aa3ed4d91174327d389b70ebd9f1f2d8e9131b3f74d7433a8a3" exitCode=0
Nov 02 13:43:29.430421 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.430399    5034 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="b409fb81a8cbb74b41f70d37cc361e8af6efa44e5b97b540271bbfdf2d47c8ac" exitCode=0
Nov 02 13:43:29.430421 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.430418    5034 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="e2d08a9a6b1c69080764d8a10d0b9198b083ab40cbc69cb3bffc6067f7a48715" exitCode=0
Nov 02 13:43:29.430535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.430425    5034 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="8ba17be7daaf2c9c44784d0975b36b82e68d539d8a4cec5d6927e90bdb1ef82b" exitCode=0
Nov 02 13:43:29.430535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.430430    5034 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="9530c571bcd9d358fdc48a4619db4282f4582c0c93e4797b8a9a83c8b7b3e7bd" exitCode=0
Nov 02 13:43:29.430535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.430435    5034 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="9798ec8ed1832291b9150443014207dcf7b91d7270fcbda38dc30b049c961ee7" exitCode=0
Nov 02 13:43:29.430535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.430441    5034 generic.go:356] "Generic (PLEG): container finished" podID="132bf469-fbc8-419c-b19a-17bcabfecce1" containerID="3b02d5b1c144cde1979c95160b5c2eb21c420f240a08e3f98346023aea532bf0" exitCode=0
Nov 02 13:43:29.445155 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.445094    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/1.log"
Nov 02 13:43:29.445155 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.445133    5034 generic.go:356] "Generic (PLEG): container finished" podID="398a3f5c-7cbe-448d-8f16-3360c18a38bf" containerID="0c97ab36897f8243ae5de4891c4454237119719e8b9ba48f3e20f4dddeb9ce3f" exitCode=255
Nov 02 13:43:29.463157 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.463143    5034 manager.go:324] Recovery completed
Nov 02 13:43:29.484269 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484246    5034 cpu_manager.go:226] "Starting" policy="none"
Nov 02 13:43:29.484269 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484260    5034 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 13:43:29.484394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484276    5034 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 13:43:29.484428 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484407    5034 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 13:43:29.484464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484425    5034 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 13:43:29.484464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484441    5034 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 13:43:29.484464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484446    5034 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 13:43:29.484464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484452    5034 policy_none.go:50] "Start"
Nov 02 13:43:29.484464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484460    5034 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 13:43:29.484604 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484468    5034 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 13:43:29.484604 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484594    5034 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 13:43:29.484604 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484602    5034 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 13:43:29.484679 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.484608    5034 policy_none.go:44] "Start"
Nov 02 13:43:29.486384 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:29.486361    5034 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 13:43:29.488906 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.488880    5034 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 13:43:29.491446 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.491431    5034 manager.go:341] "Starting Device Plugin manager"
Nov 02 13:43:29.491889 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:29.491457    5034 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 13:43:29.491889 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.491890    5034 server.go:87] "Starting device plugin registration server"
Nov 02 13:43:29.492122 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.492112    5034 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 13:43:29.492153 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.492125    5034 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 13:43:29.492256 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.492231    5034 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 13:43:29.492365 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.492339    5034 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 13:43:29.492420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.492376    5034 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 13:43:29.492420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.492382    5034 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 13:43:29.492517 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.492426    5034 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T13:43:29.492363267Z","UUID":"87f9e1b2-1371-4594-89eb-0471e81dceae","Handler":null,"Name":"","Endpoint":""}
Nov 02 13:43:29.493740 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.493725    5034 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 13:43:29.493898 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.493748    5034 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 13:43:29.494056 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:29.494028    5034 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 13:43:29.593057 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.593036    5034 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 13:43:29.594066 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.594050    5034 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 13:43:29.594135 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.594076    5034 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 13:43:29.594135 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.594086    5034 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-26.ec2.internal" event="NodeHasSufficientPID"
Nov 02 13:43:29.594135 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.594130    5034 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.629706 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.629688    5034 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.629774 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.629737    5034 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.687430 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.687387    5034 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"]
Nov 02 13:43:29.687941 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.687910    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal" event={"ID":"3694267f060f6a7f596609ec14c03ba4","Type":"ContainerStarted","Data":"b618146706c124ded121af7110a1533e01f647d5af2bb3d2711c3fc9d76d06f1"}
Nov 02 13:43:29.688043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.687944    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal" event={"ID":"3694267f060f6a7f596609ec14c03ba4","Type":"ContainerStarted","Data":"4c54c3e9ab3bcab4bbd3dc7ec62e43b33a3cfa789b1adb1cf482831ce5afa8c6"}
Nov 02 13:43:29.688043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.687969    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal" event={"ID":"7b3a0abeec74791fe4e4176d1d343021","Type":"ContainerStarted","Data":"7ec0fc4f93068ed62ec3fdb413bfb26ba9d3919f4ddaa55138e3260ac153fa4d"}
Nov 02 13:43:29.688043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.687981    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal" event={"ID":"7b3a0abeec74791fe4e4176d1d343021","Type":"ContainerDied","Data":"d1afcc17a1ae9aa3ed4d91174327d389b70ebd9f1f2d8e9131b3f74d7433a8a3"}
Nov 02 13:43:29.688043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.687994    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal" event={"ID":"7b3a0abeec74791fe4e4176d1d343021","Type":"ContainerStarted","Data":"645611df7ab78705ebae02c65af9ccd48c54ce0fe0200f88a4df8134bc435cf8"}
Nov 02 13:43:29.688043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.688004    5034 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.688435 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.688123    5034 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.714958 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.714906    5034 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:29.714958 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:29.714947    5034 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.715142 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.714961    5034 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:29.715142 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:29.715010    5034 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-26.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.803273 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.803248    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3694267f060f6a7f596609ec14c03ba4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.ec2.internal\" (UID: \"3694267f060f6a7f596609ec14c03ba4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.803359 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.803280    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.803359 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.803307    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.904034 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.904008    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.904131 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.904058    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3694267f060f6a7f596609ec14c03ba4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.ec2.internal\" (UID: \"3694267f060f6a7f596609ec14c03ba4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.904131 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.904076    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.904131 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.904078    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.904223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.904135    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7b3a0abeec74791fe4e4176d1d343021-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" (UID: \"7b3a0abeec74791fe4e4176d1d343021\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:29.904223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:29.904137    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3694267f060f6a7f596609ec14c03ba4-config\") pod \"kube-apiserver-proxy-ip-10-0-139-26.ec2.internal\" (UID: \"3694267f060f6a7f596609ec14c03ba4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:30.339352 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.339293    5034 apiserver.go:52] "Watching apiserver"
Nov 02 13:43:30.395055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.395029    5034 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 13:43:30.395715 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.395687    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4","openshift-multus/multus-additional-cni-plugins-vnt96","kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6","openshift-dns/node-resolver-t6vhd","kube-system/kas-connection-checker-685895c7c9-75dw4","kube-system/kas-connection-checker-685895c7c9-pm5xb","openshift-image-registry/image-registry-8c6787bcd-psbdk","openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n","openshift-network-diagnostics/network-check-target-2gjsj","openshift-image-registry/node-ca-29jl9","kube-system/global-pull-secret-syncer-zp7ql","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv","openshift-multus/multus-5sp5j","openshift-insights/insights-runtime-extractor-2m5tg","openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg","openshift-multus/network-metrics-daemon-g8jrw","openshift-network-console/networking-console-plugin-86c478bb88-jc4gv","openshift-image-registry/image-registry-5d555fdbf5-mnstt","openshift-service-ca/service-ca-5fffc97bdf-cggsz","kube-system/konnectivity-agent-rfknw","openshift-ovn-kubernetes/ovnkube-node-7mx2k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5tzss","openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4","openshift-ingress-canary/ingress-canary-b7nn8","openshift-console-operator/console-operator-7cd56b8dbc-zfzrk","openshift-network-operator/iptables-alerter-nsnll","openshift-insights/insights-operator-7c4986d449-f79wg","kube-system/kas-connection-checker-685895c7c9-zlk72","openshift-dns/dns-default-fzlvp","openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z","openshift-ingress/router-default-5466575dd5-gsvt5","openshift-insights/periodic-gathering-q59bv-5ptkf","openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"]
Nov 02 13:43:30.397828 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.397810    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.398208 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.398190    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:30.398604 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.398585    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.399934 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.399919    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:30.400841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.400825    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:30.408519 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.408502    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:30.409607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.409589    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.410763 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.410750    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:30.411358 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.411333    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.411931 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.411888    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.412136 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.412122    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.412136 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.412122    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 13:43:30.412912 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.412897    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 02 13:43:30.415841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.415761    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 02 13:43:30.416005 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.415993    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 02 13:43:30.416044 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416003    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 13:43:30.416284 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416270    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.416386 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416327    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.416386 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416365    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416398    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416445    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416485    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416523    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416529    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416537    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416543    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416567    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416582    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416615    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 02 13:43:30.416929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.416736    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 13:43:30.426458 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.426442    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 13:43:30.429957 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.429943    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-fs4v4\""
Nov 02 13:43:30.430036 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.429970    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 02 13:43:30.430730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.430717    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qnz8k\""
Nov 02 13:43:30.430853 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.430773    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 02 13:43:30.430853 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.430824    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 02 13:43:30.430962 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.430881    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cjxg7\""
Nov 02 13:43:30.430962 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.430906    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4wcws\""
Nov 02 13:43:30.434936 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.434911    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 13:43:30.435043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.434953    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-sk6c2\""
Nov 02 13:43:30.435043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.435007    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 13:43:30.435219 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.435089    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6lw86\""
Nov 02 13:43:30.435219 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.435143    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ffh9b\""
Nov 02 13:43:30.435219 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.435173    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9sg29\""
Nov 02 13:43:30.435394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.435327    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 13:43:30.437840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.437825    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 13:43:30.438054 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.438039    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xwmgj\""
Nov 02 13:43:30.440088 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440040    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 13:43:30.440088 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440058    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.440318 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440279    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 13:43:30.440318 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440303    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hkz22\""
Nov 02 13:43:30.440468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440327    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 02 13:43:30.440468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440330    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w4l5x\""
Nov 02 13:43:30.440468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440365    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.440468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440366    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bqvhh\""
Nov 02 13:43:30.440468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440401    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lj29d\""
Nov 02 13:43:30.440468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440449    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lvdl4\""
Nov 02 13:43:30.440468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440463    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.440875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440841    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 13:43:30.441009 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.440994    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 13:43:30.441127 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.441112    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jws44\""
Nov 02 13:43:30.441528 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.441508    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 02 13:43:30.441601 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.441591    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6wff6\""
Nov 02 13:43:30.441670 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.441654    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-n6hr6\""
Nov 02 13:43:30.441870 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.441654    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 13:43:30.441978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.441864    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-47zgd\""
Nov 02 13:43:30.442075 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.441976    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-52vg6\""
Nov 02 13:43:30.442213 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.442134    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 02 13:43:30.442370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.442272    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 13:43:30.442476 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.442385    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5dqzs\""
Nov 02 13:43:30.443657 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.443635    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 13:43:30.447696 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.447680    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.447836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.447823    5034 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:30.447927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.447911    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.447985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.447928    5034 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:30.447985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.447933    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 02 13:43:30.448261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.448247    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 02 13:43:30.448334 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.448319    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.448369 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.448349    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 13:43:30.448500 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.448490    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.448586 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.448575    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.448711 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.448700    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.448743 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.448704    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.448743 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.448726    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.449065 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.449053    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 02 13:43:30.449104 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.449075    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.451070 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.451056    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 02 13:43:30.458034 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458009    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.458529 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458501    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 13:43:30.458657 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458543    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.458657 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458550    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 13:43:30.458816 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458728    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.458917 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458886    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.458917 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458912    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.459055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458922    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 13:43:30.459055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458937    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.459055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.458962    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.459223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459055    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.459223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459088    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.459223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459095    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.459223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459102    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 02 13:43:30.459223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459119    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.459223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459133    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.459223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459150    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 13:43:30.459223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459151    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 13:43:30.459618 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459269    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.459618 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459278    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 02 13:43:30.459618 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459272    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 02 13:43:30.459618 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459270    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.459618 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459303    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.459618 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459328    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 13:43:30.459618 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.459350    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.460153 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.460136    5034 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 13:43:30.461509 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.461493    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:43:30.461600 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.461514    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 13:43:30.461600 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.461530    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 13:43:30.461600 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.461551    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 13:43:30.461600 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.461589    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 13:43:30.470340 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.470326    5034 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:30.470388 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:30.470364    5034 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-26.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:30.477180 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.477163    5034 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 13:43:30.477263 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:30.477199    5034 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-26.ec2.internal"
Nov 02 13:43:30.505606 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505585    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dr7\" (UniqueName: \"kubernetes.io/projected/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-kube-api-access-n8dr7\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.505707 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505614    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63881d4e-a3bf-4948-a369-c892baba23a8-tmp-dir\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:30.505707 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505637    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcbk\" (UniqueName: \"kubernetes.io/projected/3bfd00d2-35ee-4067-8649-bdb54ee596a2-kube-api-access-fzcbk\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:30.505707 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505665    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjt9s\" (UniqueName: \"kubernetes.io/projected/26f2db88-b241-451f-8f1e-9f8c68ad0e73-kube-api-access-sjt9s\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:43:30.505707 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505689    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.505891 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505728    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63881d4e-a3bf-4948-a369-c892baba23a8-tmp-dir\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:30.505891 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505742    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.505891 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505815    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.505891 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505848    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-kubelet-config\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:30.506074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505904    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91358478-4176-40c6-85b3-9e044fd2fa22-host-slash\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:43:30.506074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505944    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-registration-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.506074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.505999    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cnibin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.506074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506035    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.506074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506032    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:30.506228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506082    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-systemd\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.506228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506120    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63881d4e-a3bf-4948-a369-c892baba23a8-config-volume\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:30.506228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506192    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f2db88-b241-451f-8f1e-9f8c68ad0e73-tmp-dir\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:43:30.506325 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506278    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.506325 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506288    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26f2db88-b241-451f-8f1e-9f8c68ad0e73-tmp-dir\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:43:30.506325 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506306    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d26dd5a-118a-4d50-9661-22feeb24d158-agent-certs\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:43:30.506470 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506340    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-system-cni-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.506470 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506340    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63881d4e-a3bf-4948-a369-c892baba23a8-config-volume\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:30.506470 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506384    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk7vv\" (UniqueName: \"kubernetes.io/projected/4150923a-78f1-4f52-a0d5-271ec942bd1b-kube-api-access-jk7vv\") pod \"volume-data-source-validator-58c9f5f46d-np42z\" (UID: \"4150923a-78f1-4f52-a0d5-271ec942bd1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"
Nov 02 13:43:30.506614 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506507    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-systemd-units\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.506614 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506555    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:30.506614 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506590    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d26dd5a-118a-4d50-9661-22feeb24d158-agent-certs\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:43:30.506614 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506607    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.506762 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506609    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb92t\" (UniqueName: \"kubernetes.io/projected/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-api-access-hb92t\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.506762 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506655    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhqc\" (UniqueName: \"kubernetes.io/projected/1859de72-db9b-4e60-ac47-b8360156ed1a-kube-api-access-qfhqc\") pod \"migrator-66df5d496b-697rr\" (UID: \"1859de72-db9b-4e60-ac47-b8360156ed1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"
Nov 02 13:43:30.506762 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506681    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.506762 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506706    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-default-certificate\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.506762 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506628    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:30.506762 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506731    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.506762 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506758    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506810    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-sys\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506852    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506898    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-env-overrides\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506925    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506948    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-default-certificate\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506955    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-daemon-config\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.506990    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507016    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8646ac2d-cfd2-4b7c-929a-906815e6551f-serving-cert\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507041    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-socket-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507067    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovn-node-metrics-cert\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507092    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-kubelet\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507095    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.507120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507118    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrph\" (UniqueName: \"kubernetes.io/projected/132bf469-fbc8-419c-b19a-17bcabfecce1-kube-api-access-hsrph\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507140    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507144    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhlq\" (UniqueName: \"kubernetes.io/projected/35f03797-afb7-4aa5-a841-c665e99feaa4-kube-api-access-tzhlq\") pod \"kas-connection-checker-685895c7c9-zlk72\" (UID: \"35f03797-afb7-4aa5-a841-c665e99feaa4\") " pod="kube-system/kas-connection-checker-685895c7c9-zlk72"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507187    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507214    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cni-binary-copy\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507233    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-env-overrides\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507239    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-snapshots\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507261    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-slash\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507343    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-snapshots\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507347    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-daemon-config\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507431    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cni-binary-copy\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507489    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8646ac2d-cfd2-4b7c-929a-906815e6551f-serving-cert\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507492    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507541    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507571    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-host\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507598    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507641    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v5tx\" (UniqueName: \"kubernetes.io/projected/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-kube-api-access-4v5tx\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507669    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27e3e0a4-dc29-4e68-9344-f82ea07cb067-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:30.507701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507693    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-trusted-ca\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507715    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-lib-modules\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507735    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovn-node-metrics-cert\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507740    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507809    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00be7ba2-6f06-4ed5-a007-d0ac612177ee-original-pull-secret\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507814    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507845    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-bin\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507875    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507906    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqql\" (UniqueName: \"kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507924    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507930    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.507992    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/8816a23c-15b8-4775-8853-6dc6cf7526c5-kube-api-access-qjlx2\") pod \"kas-connection-checker-685895c7c9-75dw4\" (UID: \"8816a23c-15b8-4775-8853-6dc6cf7526c5\") " pod="kube-system/kas-connection-checker-685895c7c9-75dw4"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508022    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508038    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508052    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508064    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-var-lib-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508175    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-trusted-ca\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508219    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-k8s-cni-cncf-io\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508250    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-etc-kubernetes\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508255    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27e3e0a4-dc29-4e68-9344-f82ea07cb067-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508292    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9f7\" (UniqueName: \"kubernetes.io/projected/91358478-4176-40c6-85b3-9e044fd2fa22-kube-api-access-cc9f7\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508326    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508353    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-kubernetes\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508376    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-netd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508441    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508474    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27e3e0a4-dc29-4e68-9344-f82ea07cb067-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508500    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508518    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508532    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrcq\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508559    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpj5\" (UniqueName: \"kubernetes.io/projected/8646ac2d-cfd2-4b7c-929a-906815e6551f-kube-api-access-dkpj5\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508583    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-netns\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.508607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508608    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-systemd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508637    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-script-lib\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508663    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rlg\" (UniqueName: \"kubernetes.io/projected/91600010-0a54-4517-b9a7-1cc38b075abd-kube-api-access-47rlg\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508689    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4fv4\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508713    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508719    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508738    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xskc\" (UniqueName: \"kubernetes.io/projected/d861fac3-737a-4cd7-abc6-4af75b4c1d67-kube-api-access-5xskc\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508763    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-hostroot\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508768    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/27e3e0a4-dc29-4e68-9344-f82ea07cb067-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508802    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-etc-tuned\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508828    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-stats-auth\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.508883    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-kubelet\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509003    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509019    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-script-lib\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509035    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-socket-dir-parent\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509076    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-binary-copy\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509079    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-etc-tuned\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509093    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509141    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-stats-auth\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509178    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-etc-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509267    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-ovn\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509289    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509296    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxts2\" (UniqueName: \"kubernetes.io/projected/398a3f5c-7cbe-448d-8f16-3360c18a38bf-kube-api-access-xxts2\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509332    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zld\" (UniqueName: \"kubernetes.io/projected/df16fea1-6bca-47f3-b9bd-35716e3d50f3-kube-api-access-v9zld\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509352    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-log-socket\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509367    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509372    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4p82\" (UniqueName: \"kubernetes.io/projected/63881d4e-a3bf-4948-a369-c892baba23a8-kube-api-access-f4p82\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509544    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f2db88-b241-451f-8f1e-9f8c68ad0e73-hosts-file\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509568    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-dbus\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509585    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509610    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrlq\" (UniqueName: \"kubernetes.io/projected/18d90fc8-23b1-4fca-a9e2-83eb8f4786d4-kube-api-access-qdrlq\") pod \"kas-connection-checker-685895c7c9-pm5xb\" (UID: \"18d90fc8-23b1-4fca-a9e2-83eb8f4786d4\") " pod="kube-system/kas-connection-checker-685895c7c9-pm5xb"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509759    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-conf\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509802    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509817    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-host\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509834    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509857    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bx75\" (UniqueName: \"kubernetes.io/projected/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-kube-api-access-2bx75\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:43:30.509919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509872    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.509973    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysconfig\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510065    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-service-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510090    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510108    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/028f7fde-fe8a-4d88-9c6c-54193c5c490b-crio-socket\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510123    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-device-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510149    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2ca56ffb-997f-4edb-9efa-0609042e8f08-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510151    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510191    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df16fea1-6bca-47f3-b9bd-35716e3d50f3-cert\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510219    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-key\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510266    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs7rc\" (UniqueName: \"kubernetes.io/projected/488bddc1-06e8-446c-b612-5ddeac1d8bdf-kube-api-access-rs7rc\") pod \"network-check-source-6f78d59f99-nkjt5\" (UID: \"488bddc1-06e8-446c-b612-5ddeac1d8bdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510293    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-system-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510357    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510400    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-conf-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510454    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2ca56ffb-997f-4edb-9efa-0609042e8f08-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510454    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-key\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510457    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb81289b-9a76-4529-a6f5-649ea7c85d67-config\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510492    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/df16fea1-6bca-47f3-b9bd-35716e3d50f3-cert\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510512    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vw64\" (UniqueName: \"kubernetes.io/projected/2ca56ffb-997f-4edb-9efa-0609042e8f08-kube-api-access-8vw64\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510538    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-tmp\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510561    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510608    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-serviceca\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510617    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8646ac2d-cfd2-4b7c-929a-906815e6551f-service-ca-bundle\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510626    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb81289b-9a76-4529-a6f5-649ea7c85d67-config\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510620    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4366155b-b272-4b13-b328-4d9f87325b84-tmp\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510649    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-netns\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510675    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-multus-certs\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510703    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb81289b-9a76-4529-a6f5-649ea7c85d67-serving-cert\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510734    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-modprobe-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510735    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510810    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-serviceca\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510850    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-cnibin\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510870    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-config\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510909    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhb7\" (UniqueName: \"kubernetes.io/projected/ea36ca35-7e60-41a3-b25a-0e39965f22bf-kube-api-access-srhb7\") pod \"migrator-66df5d496b-wn9j4\" (UID: \"ea36ca35-7e60-41a3-b25a-0e39965f22bf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.510956    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511006    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/028f7fde-fe8a-4d88-9c6c-54193c5c490b-data-volume\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511017    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-ovnkube-config\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511033    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb81289b-9a76-4529-a6f5-649ea7c85d67-serving-cert\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511048    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4fcq\" (UniqueName: \"kubernetes.io/projected/3b44af93-ab31-483d-9eab-3e996c90eee7-kube-api-access-d4fcq\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511081    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-var-lib-kubelet\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511108    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91358478-4176-40c6-85b3-9e044fd2fa22-iptables-alerter-script\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511137    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511163    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511200    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/028f7fde-fe8a-4d88-9c6c-54193c5c490b-data-volume\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511209    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511326    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511345    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-run\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.511373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511382    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-bin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511384    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/91358478-4176-40c6-85b3-9e044fd2fa22-iptables-alerter-script\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511417    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511451    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/398a3f5c-7cbe-448d-8f16-3360c18a38bf-serving-cert\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511476    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b34b2c6f-210f-4256-8383-16f6a3a360bb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511496    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d26dd5a-118a-4d50-9661-22feeb24d158-konnectivity-ca\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511598    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ktl\" (UniqueName: \"kubernetes.io/projected/4366155b-b272-4b13-b328-4d9f87325b84-kube-api-access-98ktl\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511626    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/398a3f5c-7cbe-448d-8f16-3360c18a38bf-serving-cert\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511648    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-tmp\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511658    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d26dd5a-118a-4d50-9661-22feeb24d158-konnectivity-ca\") pod \"konnectivity-agent-rfknw\" (UID: \"5d26dd5a-118a-4d50-9661-22feeb24d158\") " pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511662    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b34b2c6f-210f-4256-8383-16f6a3a360bb-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511747    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-multus\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511797    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-config\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511831    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fv5\" (UniqueName: \"kubernetes.io/projected/0fb4e264-b7b8-46de-b222-25331e6400c4-kube-api-access-g4fv5\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511850    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8646ac2d-cfd2-4b7c-929a-906815e6551f-tmp\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511926    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511950    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511983    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/398a3f5c-7cbe-448d-8f16-3360c18a38bf-config\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.511996    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-sys-fs\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.512035    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-cabundle\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.512103    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-node-log\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.512137    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58czq\" (UniqueName: \"kubernetes.io/projected/27e3e0a4-dc29-4e68-9344-f82ea07cb067-kube-api-access-58czq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.512169    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhvr\" (UniqueName: \"kubernetes.io/projected/fb81289b-9a76-4529-a6f5-649ea7c85d67-kube-api-access-vlhvr\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.512174    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0fb4e264-b7b8-46de-b222-25331e6400c4-signing-cabundle\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.512185    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.512731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.512367    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/132bf469-fbc8-419c-b19a-17bcabfecce1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.539302 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.539264    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dr7\" (UniqueName: \"kubernetes.io/projected/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-kube-api-access-n8dr7\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.539375 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.539330    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhlq\" (UniqueName: \"kubernetes.io/projected/35f03797-afb7-4aa5-a841-c665e99feaa4-kube-api-access-tzhlq\") pod \"kas-connection-checker-685895c7c9-zlk72\" (UID: \"35f03797-afb7-4aa5-a841-c665e99feaa4\") " pod="kube-system/kas-connection-checker-685895c7c9-zlk72"
Nov 02 13:43:30.539493 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.539478    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjt9s\" (UniqueName: \"kubernetes.io/projected/26f2db88-b241-451f-8f1e-9f8c68ad0e73-kube-api-access-sjt9s\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:43:30.539948 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.539929    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcbk\" (UniqueName: \"kubernetes.io/projected/3bfd00d2-35ee-4067-8649-bdb54ee596a2-kube-api-access-fzcbk\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:30.540416 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.540401    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk7vv\" (UniqueName: \"kubernetes.io/projected/4150923a-78f1-4f52-a0d5-271ec942bd1b-kube-api-access-jk7vv\") pod \"volume-data-source-validator-58c9f5f46d-np42z\" (UID: \"4150923a-78f1-4f52-a0d5-271ec942bd1b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-np42z"
Nov 02 13:43:30.542727 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.542704    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.542857 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.542752    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9f7\" (UniqueName: \"kubernetes.io/projected/91358478-4176-40c6-85b3-9e044fd2fa22-kube-api-access-cc9f7\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:43:30.543056 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.543040    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb92t\" (UniqueName: \"kubernetes.io/projected/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-api-access-hb92t\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.544691 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.544674    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v5tx\" (UniqueName: \"kubernetes.io/projected/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-kube-api-access-4v5tx\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.545069 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.545049    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vrz\" (UniqueName: \"kubernetes.io/projected/fb318d8f-e7de-4185-96cd-d592b304248a-kube-api-access-47vrz\") pod \"network-check-target-2gjsj\" (UID: \"fb318d8f-e7de-4185-96cd-d592b304248a\") " pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:30.545216 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.545202    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqql\" (UniqueName: \"kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql\") pod \"periodic-gathering-q59bv-5ptkf\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") " pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:43:30.550049 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.550029    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrph\" (UniqueName: \"kubernetes.io/projected/132bf469-fbc8-419c-b19a-17bcabfecce1-kube-api-access-hsrph\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.554414 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.554394    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rlg\" (UniqueName: \"kubernetes.io/projected/91600010-0a54-4517-b9a7-1cc38b075abd-kube-api-access-47rlg\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:30.554616 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.554580    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/8816a23c-15b8-4775-8853-6dc6cf7526c5-kube-api-access-qjlx2\") pod \"kas-connection-checker-685895c7c9-75dw4\" (UID: \"8816a23c-15b8-4775-8853-6dc6cf7526c5\") " pod="kube-system/kas-connection-checker-685895c7c9-75dw4"
Nov 02 13:43:30.555343 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.555291    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zld\" (UniqueName: \"kubernetes.io/projected/df16fea1-6bca-47f3-b9bd-35716e3d50f3-kube-api-access-v9zld\") pod \"ingress-canary-b7nn8\" (UID: \"df16fea1-6bca-47f3-b9bd-35716e3d50f3\") " pod="openshift-ingress-canary/ingress-canary-b7nn8"
Nov 02 13:43:30.555343 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.555304    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhb7\" (UniqueName: \"kubernetes.io/projected/ea36ca35-7e60-41a3-b25a-0e39965f22bf-kube-api-access-srhb7\") pod \"migrator-66df5d496b-wn9j4\" (UID: \"ea36ca35-7e60-41a3-b25a-0e39965f22bf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-wn9j4"
Nov 02 13:43:30.555463 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.555400    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vw64\" (UniqueName: \"kubernetes.io/projected/2ca56ffb-997f-4edb-9efa-0609042e8f08-kube-api-access-8vw64\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:30.555501 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.555463    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpj5\" (UniqueName: \"kubernetes.io/projected/8646ac2d-cfd2-4b7c-929a-906815e6551f-kube-api-access-dkpj5\") pod \"insights-operator-7c4986d449-f79wg\" (UID: \"8646ac2d-cfd2-4b7c-929a-906815e6551f\") " pod="openshift-insights/insights-operator-7c4986d449-f79wg"
Nov 02 13:43:30.555580 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.555563    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bx75\" (UniqueName: \"kubernetes.io/projected/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-kube-api-access-2bx75\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:43:30.555620 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.555575    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrlq\" (UniqueName: \"kubernetes.io/projected/18d90fc8-23b1-4fca-a9e2-83eb8f4786d4-kube-api-access-qdrlq\") pod \"kas-connection-checker-685895c7c9-pm5xb\" (UID: \"18d90fc8-23b1-4fca-a9e2-83eb8f4786d4\") " pod="kube-system/kas-connection-checker-685895c7c9-pm5xb"
Nov 02 13:43:30.555651 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.555622    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxts2\" (UniqueName: \"kubernetes.io/projected/398a3f5c-7cbe-448d-8f16-3360c18a38bf-kube-api-access-xxts2\") pod \"console-operator-7cd56b8dbc-zfzrk\" (UID: \"398a3f5c-7cbe-448d-8f16-3360c18a38bf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:30.555763 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.555749    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs7rc\" (UniqueName: \"kubernetes.io/projected/488bddc1-06e8-446c-b612-5ddeac1d8bdf-kube-api-access-rs7rc\") pod \"network-check-source-6f78d59f99-nkjt5\" (UID: \"488bddc1-06e8-446c-b612-5ddeac1d8bdf\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-nkjt5"
Nov 02 13:43:30.556196 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.556180    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4fcq\" (UniqueName: \"kubernetes.io/projected/3b44af93-ab31-483d-9eab-3e996c90eee7-kube-api-access-d4fcq\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.556363 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.556342    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhqc\" (UniqueName: \"kubernetes.io/projected/1859de72-db9b-4e60-ac47-b8360156ed1a-kube-api-access-qfhqc\") pod \"migrator-66df5d496b-697rr\" (UID: \"1859de72-db9b-4e60-ac47-b8360156ed1a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-697rr"
Nov 02 13:43:30.556520 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.556505    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4fv4\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.560003 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.559983    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrcq\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.560096 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.560074    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xskc\" (UniqueName: \"kubernetes.io/projected/d861fac3-737a-4cd7-abc6-4af75b4c1d67-kube-api-access-5xskc\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.560135 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.560093    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhvr\" (UniqueName: \"kubernetes.io/projected/fb81289b-9a76-4529-a6f5-649ea7c85d67-kube-api-access-vlhvr\") pod \"service-ca-operator-544d749c8c-mtl2n\" (UID: \"fb81289b-9a76-4529-a6f5-649ea7c85d67\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n"
Nov 02 13:43:30.560241 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.560224    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58czq\" (UniqueName: \"kubernetes.io/projected/27e3e0a4-dc29-4e68-9344-f82ea07cb067-kube-api-access-58czq\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6\" (UID: \"27e3e0a4-dc29-4e68-9344-f82ea07cb067\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6"
Nov 02 13:43:30.562895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.562879    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.563001 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.562975    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ktl\" (UniqueName: \"kubernetes.io/projected/4366155b-b272-4b13-b328-4d9f87325b84-kube-api-access-98ktl\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.563239 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.563224    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4p82\" (UniqueName: \"kubernetes.io/projected/63881d4e-a3bf-4948-a369-c892baba23a8-kube-api-access-f4p82\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:30.563376 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.563362    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fv5\" (UniqueName: \"kubernetes.io/projected/0fb4e264-b7b8-46de-b222-25331e6400c4-kube-api-access-g4fv5\") pod \"service-ca-5fffc97bdf-cggsz\" (UID: \"0fb4e264-b7b8-46de-b222-25331e6400c4\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cggsz"
Nov 02 13:43:30.612461 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612418    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-conf-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612461 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612442    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-netns\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612459    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-multus-certs\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612484    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-modprobe-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.612585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612487    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-conf-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612506    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-netns\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612518    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-multus-certs\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612507    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-cnibin\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.612585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612540    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-cnibin\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.612585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612573    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-modprobe-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.612585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612563    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612617    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-var-lib-kubelet\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612648    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612671    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-run\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612692    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-bin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612717    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612738    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-run\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612747    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-multus\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612766    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-var-lib-kubelet\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612770    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-bin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612773    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612822    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612829    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-cni-multus\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612846    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612838    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-sys-fs\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612875    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-sys-fs\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612892    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-node-log\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612926    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612954    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-kubelet-config\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612959    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-node-log\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.612987 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:30.612958    5034 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:30.613649 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613054    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-d\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.613649 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.612986    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91358478-4176-40c6-85b3-9e044fd2fa22-host-slash\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:43:30.613649 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613637    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-registration-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.613802 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:30.613677    5034 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:43:30.613802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613052    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-kubelet-config\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:30.613802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613127    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/91358478-4176-40c6-85b3-9e044fd2fa22-host-slash\") pod \"iptables-alerter-nsnll\" (UID: \"91358478-4176-40c6-85b3-9e044fd2fa22\") " pod="openshift-network-operator/iptables-alerter-nsnll"
Nov 02 13:43:30.613802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613677    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cnibin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.613802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613733    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-cnibin\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.613802 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:30.613759    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:43:31.113739291 +0000 UTC m=+1.865043723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : secret "metrics-daemon-secret" not found
Nov 02 13:43:30.613802 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:30.613800    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:31.113767522 +0000 UTC m=+1.865071952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613810    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-registration-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613808    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613828    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/028f7fde-fe8a-4d88-9c6c-54193c5c490b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613851    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-systemd\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613890    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-system-cni-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613925    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-systemd-units\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613952    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613952    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-systemd\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.613986    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/132bf469-fbc8-419c-b19a-17bcabfecce1-system-cni-dir\") pod \"multus-additional-cni-plugins-vnt96\" (UID: \"132bf469-fbc8-419c-b19a-17bcabfecce1\") " pod="openshift-multus/multus-additional-cni-plugins-vnt96"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614001    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-systemd-units\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614024    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614025    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614054    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614084    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614042    5034 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 13:43:30.614118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614116    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-sys\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614147    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614176    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-socket-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614203    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-sys\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614206    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-kubelet\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614259    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-var-lib-kubelet\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614281    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614291    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-etc-selinux\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614321    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-slash\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614334    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-socket-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614345    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614363    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-host\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614386    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-slash\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614390    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614441    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-lib-modules\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614467    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-bin\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614497    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614527    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614550    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-lib-modules\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614560    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614596    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-var-lib-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614630    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-k8s-cni-cncf-io\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614660    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-etc-kubernetes\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614692    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-kubernetes\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614809    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-host\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.614836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614822    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-netd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614873    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-netd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614880    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614929    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-netns\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614965    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-systemd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.614982    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-host-run-k8s-cni-cncf-io\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615007    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-hostroot\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615051    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-kubelet\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615095    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-socket-dir-parent\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615128    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-hostroot\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615186    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-etc-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615217    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-var-lib-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615234    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-etc-kubernetes\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615269    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-kubelet\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615297    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-kubernetes\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615336    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-socket-dir-parent\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615352    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-run-netns\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615389    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-systemd\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615387    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615130    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-etc-openvswitch\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615453    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-ovn\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615456    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-host-cni-bin\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615515    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-multus-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615528    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-run-ovn\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615550    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-log-socket\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615585    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f2db88-b241-451f-8f1e-9f8c68ad0e73-hosts-file\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615616    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-dbus\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615644    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f46f80e0-332a-4c19-81a2-08cbb65f7fb8-log-socket\") pod \"ovnkube-node-7mx2k\" (UID: \"f46f80e0-332a-4c19-81a2-08cbb65f7fb8\") " pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615647    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-conf\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615689    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615720    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-host\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615757    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615768    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysctl-conf\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615804    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysconfig\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615839    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/028f7fde-fe8a-4d88-9c6c-54193c5c490b-crio-socket\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615848    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26f2db88-b241-451f-8f1e-9f8c68ad0e73-hosts-file\") pod \"node-resolver-t6vhd\" (UID: \"26f2db88-b241-451f-8f1e-9f8c68ad0e73\") " pod="openshift-dns/node-resolver-t6vhd"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615865    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-device-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615901    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-system-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.615920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.615911    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00be7ba2-6f06-4ed5-a007-d0ac612177ee-dbus\") pod \"global-pull-secret-syncer-zp7ql\" (UID: \"00be7ba2-6f06-4ed5-a007-d0ac612177ee\") " pod="kube-system/global-pull-secret-syncer-zp7ql"
Nov 02 13:43:30.617535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.616017    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd-host\") pod \"node-ca-29jl9\" (UID: \"c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd\") " pod="openshift-image-registry/node-ca-29jl9"
Nov 02 13:43:30.617535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.616140    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b44af93-ab31-483d-9eab-3e996c90eee7-device-dir\") pod \"aws-ebs-csi-driver-node-7jfmv\" (UID: \"3b44af93-ab31-483d-9eab-3e996c90eee7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7jfmv"
Nov 02 13:43:30.617535 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:30.616204    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:31.116183473 +0000 UTC m=+1.867487909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : configmap references non-existent config key: service-ca.crt
Nov 02 13:43:30.617535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.616202    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4366155b-b272-4b13-b328-4d9f87325b84-etc-sysconfig\") pod \"tuned-5tzss\" (UID: \"4366155b-b272-4b13-b328-4d9f87325b84\") " pod="openshift-cluster-node-tuning-operator/tuned-5tzss"
Nov 02 13:43:30.617535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.616255    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/028f7fde-fe8a-4d88-9c6c-54193c5c490b-crio-socket\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:30.617535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.616287    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d67e5a-851c-4fdd-befa-c0a97e93d5d2-system-cni-dir\") pod \"multus-5sp5j\" (UID: \"c2d67e5a-851c-4fdd-befa-c0a97e93d5d2\") " pod="openshift-multus/multus-5sp5j"
Nov 02 13:43:30.617535 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:30.616380    5034 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:43:30.617535 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:30.616439    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls podName:63881d4e-a3bf-4948-a369-c892baba23a8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:31.11642429 +0000 UTC m=+1.867728721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls") pod "dns-default-fzlvp" (UID: "63881d4e-a3bf-4948-a369-c892baba23a8") : secret "dns-default-metrics-tls" not found
Nov 02 13:43:30.617535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.617157    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2ca56ffb-997f-4edb-9efa-0609042e8f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-nnxtg\" (UID: \"2ca56ffb-997f-4edb-9efa-0609042e8f08\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:30.619842 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.619189    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"image-registry-8c6787bcd-psbdk\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") " pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.621156 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.620773    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"image-registry-5d555fdbf5-mnstt\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") " pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.621156 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.620985    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d861fac3-737a-4cd7-abc6-4af75b4c1d67-metrics-certs\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:30.621156 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.621126    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b34b2c6f-210f-4256-8383-16f6a3a360bb-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jc4gv\" (UID: \"b34b2c6f-210f-4256-8383-16f6a3a360bb\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:30.622080 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.622058    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3bfd00d2-35ee-4067-8649-bdb54ee596a2-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5ssg4\" (UID: \"3bfd00d2-35ee-4067-8649-bdb54ee596a2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:30.698500 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.698483    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"
Nov 02 13:43:30.699574 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.699546    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:30.700639 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.700617    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"
Nov 02 13:43:30.701726 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.701714    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"
Nov 02 13:43:30.701806 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.701766    5034 scope.go:122] "RemoveContainer" containerID="0c97ab36897f8243ae5de4891c4454237119719e8b9ba48f3e20f4dddeb9ce3f"
Nov 02 13:43:30.710872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.710853    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:30.879808 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:30.879624    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t6vhd_26f2db88-b241-451f-8f1e-9f8c68ad0e73/dns-node-resolver/0.log"
Nov 02 13:43:31.046272 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.046245    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4"]
Nov 02 13:43:31.053925 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.053910    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d555fdbf5-mnstt"]
Nov 02 13:43:31.061976 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:31.061919    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa80f89_f24d_4f86_940c_864dbf177a5c.slice/crio-901a0f1565ded0308d83487d185d54a54fbb53c714cd3c8ba3df90d291cf0b7d WatchSource:0}: Error finding container 901a0f1565ded0308d83487d185d54a54fbb53c714cd3c8ba3df90d291cf0b7d: Status 404 returned error can't find the container with id 901a0f1565ded0308d83487d185d54a54fbb53c714cd3c8ba3df90d291cf0b7d
Nov 02 13:43:31.117331 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.117309    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg"]
Nov 02 13:43:31.118244 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:31.118207    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca56ffb_997f_4edb_9efa_0609042e8f08.slice/crio-63261cd772398a5eb8dcc697353d463089c6647277d2b742338a65c4afa09cb8 WatchSource:0}: Error finding container 63261cd772398a5eb8dcc697353d463089c6647277d2b742338a65c4afa09cb8: Status 404 returned error can't find the container with id 63261cd772398a5eb8dcc697353d463089c6647277d2b742338a65c4afa09cb8
Nov 02 13:43:31.120000 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.119466    5034 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:43:31.120161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.120138    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:31.120245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.120180    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:31.120245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.120213    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:31.120245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.120235    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:31.120380 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:31.120320    5034 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:31.120380 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:31.120356    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:32.120344544 +0000 UTC m=+2.871648960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:31.120503 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:31.120389    5034 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:43:31.120503 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:31.120409    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls podName:63881d4e-a3bf-4948-a369-c892baba23a8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:32.120403302 +0000 UTC m=+2.871707718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls") pod "dns-default-fzlvp" (UID: "63881d4e-a3bf-4948-a369-c892baba23a8") : secret "dns-default-metrics-tls" not found
Nov 02 13:43:31.120503 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:31.120454    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:32.120447493 +0000 UTC m=+2.871751909 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : configmap references non-existent config key: service-ca.crt
Nov 02 13:43:31.120503 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:31.120488    5034 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:43:31.120715 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:31.120507    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:43:32.120501058 +0000 UTC m=+2.871805474 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : secret "metrics-daemon-secret" not found
Nov 02 13:43:31.121874 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.121797    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jc4gv"]
Nov 02 13:43:31.123597 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:31.123580    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34b2c6f_210f_4256_8383_16f6a3a360bb.slice/crio-e5aac71756ecb1c8587b17873023c2d2e1ad14e1b6cf1a35644bd67f833a076f WatchSource:0}: Error finding container e5aac71756ecb1c8587b17873023c2d2e1ad14e1b6cf1a35644bd67f833a076f: Status 404 returned error can't find the container with id e5aac71756ecb1c8587b17873023c2d2e1ad14e1b6cf1a35644bd67f833a076f
Nov 02 13:43:31.137014 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.136993    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8c6787bcd-psbdk"]
Nov 02 13:43:31.137213 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:31.137192    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633c4858_9dc5_4049_aec3_8a14171652ae.slice/crio-0b02c25bf9c1d0684da7a1cee4f2d42fd7607b285fb8570e3332f8ea6d4e5ac9 WatchSource:0}: Error finding container 0b02c25bf9c1d0684da7a1cee4f2d42fd7607b285fb8570e3332f8ea6d4e5ac9: Status 404 returned error can't find the container with id 0b02c25bf9c1d0684da7a1cee4f2d42fd7607b285fb8570e3332f8ea6d4e5ac9
Nov 02 13:43:31.458561 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.458475    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt" event={"ID":"3fa80f89-f24d-4f86-940c-864dbf177a5c","Type":"ContainerStarted","Data":"3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5"}
Nov 02 13:43:31.459094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.458572    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt" event={"ID":"3fa80f89-f24d-4f86-940c-864dbf177a5c","Type":"ContainerStarted","Data":"901a0f1565ded0308d83487d185d54a54fbb53c714cd3c8ba3df90d291cf0b7d"}
Nov 02 13:43:31.459889 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.459861    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv" event={"ID":"b34b2c6f-210f-4256-8383-16f6a3a360bb","Type":"ContainerStarted","Data":"e5aac71756ecb1c8587b17873023c2d2e1ad14e1b6cf1a35644bd67f833a076f"}
Nov 02 13:43:31.461190 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.461161    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4" event={"ID":"3bfd00d2-35ee-4067-8649-bdb54ee596a2","Type":"ContainerStarted","Data":"e8a5b33377a751003f3b6c51bfd782316ff097e253b0c6703053b8729e50403c"}
Nov 02 13:43:31.462760 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.462739    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 13:43:31.463174 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.463157    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/1.log"
Nov 02 13:43:31.463256 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.463191    5034 generic.go:356] "Generic (PLEG): container finished" podID="398a3f5c-7cbe-448d-8f16-3360c18a38bf" containerID="9edd8c64cb2d5f3842349b869336e15eb19b2e89ba5407353efbef31430d160e" exitCode=255
Nov 02 13:43:31.463316 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.463269    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" event={"ID":"398a3f5c-7cbe-448d-8f16-3360c18a38bf","Type":"ContainerDied","Data":"9edd8c64cb2d5f3842349b869336e15eb19b2e89ba5407353efbef31430d160e"}
Nov 02 13:43:31.463377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.463327    5034 scope.go:122] "RemoveContainer" containerID="0c97ab36897f8243ae5de4891c4454237119719e8b9ba48f3e20f4dddeb9ce3f"
Nov 02 13:43:31.463593 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.463574    5034 scope.go:122] "RemoveContainer" containerID="9edd8c64cb2d5f3842349b869336e15eb19b2e89ba5407353efbef31430d160e"
Nov 02 13:43:31.463840 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:31.463813    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zfzrk_openshift-console-operator(398a3f5c-7cbe-448d-8f16-3360c18a38bf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" podUID="398a3f5c-7cbe-448d-8f16-3360c18a38bf"
Nov 02 13:43:31.464917 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.464894    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg" event={"ID":"2ca56ffb-997f-4edb-9efa-0609042e8f08","Type":"ContainerStarted","Data":"63261cd772398a5eb8dcc697353d463089c6647277d2b742338a65c4afa09cb8"}
Nov 02 13:43:31.467207 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.466999    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk" event={"ID":"633c4858-9dc5-4049-aec3-8a14171652ae","Type":"ContainerStarted","Data":"bd87bffa132f9b5504404f362f74d971cc572ed005070825494acce7ac91cdd4"}
Nov 02 13:43:31.467207 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.467043    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk" event={"ID":"633c4858-9dc5-4049-aec3-8a14171652ae","Type":"ContainerStarted","Data":"0b02c25bf9c1d0684da7a1cee4f2d42fd7607b285fb8570e3332f8ea6d4e5ac9"}
Nov 02 13:43:31.478503 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.478484    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-29jl9_c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd/node-ca/0.log"
Nov 02 13:43:31.497856 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.497833    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:43:31.948628 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:31.948599    5034 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.129886    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.129941    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.130019    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.130088    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:32.130234    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.130213134 +0000 UTC m=+4.881517568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : configmap references non-existent config key: service-ca.crt
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:32.130321    5034 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:32.130355    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.130343621 +0000 UTC m=+4.881648050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : secret "metrics-daemon-secret" not found
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:32.130442    5034 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:32.130546    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.130470859 +0000 UTC m=+4.881775277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:32.130607    5034 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 13:43:32.131041 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:32.130641    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls podName:63881d4e-a3bf-4948-a369-c892baba23a8 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:34.130629593 +0000 UTC m=+4.881934016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls") pod "dns-default-fzlvp" (UID: "63881d4e-a3bf-4948-a369-c892baba23a8") : secret "dns-default-metrics-tls" not found
Nov 02 13:43:32.278147 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.278120    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b7nn8_df16fea1-6bca-47f3-b9bd-35716e3d50f3/serve-healthcheck-canary/0.log"
Nov 02 13:43:32.473169 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.473125    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jc4gv" event={"ID":"b34b2c6f-210f-4256-8383-16f6a3a360bb","Type":"ContainerStarted","Data":"fb5a31f9ba8edf48054369526ed4674089f6a9c79bf31c8bda88887c2c4f17fa"}
Nov 02 13:43:32.477043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.474996    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 13:43:32.477043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.475704    5034 scope.go:122] "RemoveContainer" containerID="9edd8c64cb2d5f3842349b869336e15eb19b2e89ba5407353efbef31430d160e"
Nov 02 13:43:32.477043 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:32.475881    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zfzrk_openshift-console-operator(398a3f5c-7cbe-448d-8f16-3360c18a38bf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" podUID="398a3f5c-7cbe-448d-8f16-3360c18a38bf"
Nov 02 13:43:32.477043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.476202    5034 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:32.478103 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.478082    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-697rr_1859de72-db9b-4e60-ac47-b8360156ed1a/migrator/0.log"
Nov 02 13:43:32.568152 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.567294    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:32.677949 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.677918    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-wn9j4_ea36ca35-7e60-41a3-b25a-0e39965f22bf/migrator/0.log"
Nov 02 13:43:32.883029 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:32.882953    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6_27e3e0a4-dc29-4e68-9344-f82ea07cb067/kube-storage-version-migrator-operator/0.log"
Nov 02 13:43:33.480089 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:33.480015    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4" event={"ID":"3bfd00d2-35ee-4067-8649-bdb54ee596a2","Type":"ContainerStarted","Data":"2b6a1757a35850067c139d7e0988e416f6f52fdef3eebf9916d181fac66575f1"}
Nov 02 13:43:33.480089 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:33.480072    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5ssg4" event={"ID":"3bfd00d2-35ee-4067-8649-bdb54ee596a2","Type":"ContainerStarted","Data":"79aa4aa5f584efe6a7ee620744e5d0e8eba4871dbb7e8e2139af2831ab30d4d3"}
Nov 02 13:43:33.481626 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:33.481601    5034 generic.go:356] "Generic (PLEG): container finished" podID="8646ac2d-cfd2-4b7c-929a-906815e6551f" containerID="aafbeb05fba56682112c11251207e14667fb860ac195b82aa9d7d61bf56a15a6" exitCode=0
Nov 02 13:43:33.481722 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:33.481673    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-f79wg" event={"ID":"8646ac2d-cfd2-4b7c-929a-906815e6551f","Type":"ContainerDied","Data":"aafbeb05fba56682112c11251207e14667fb860ac195b82aa9d7d61bf56a15a6"}
Nov 02 13:43:33.481985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:33.481971    5034 scope.go:122] "RemoveContainer" containerID="aafbeb05fba56682112c11251207e14667fb860ac195b82aa9d7d61bf56a15a6"
Nov 02 13:43:33.484588 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:33.484564    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-nnxtg" event={"ID":"2ca56ffb-997f-4edb-9efa-0609042e8f08","Type":"ContainerStarted","Data":"8b7b65246b72ce647b87781999ffba0896ede51ceee7cde4b4121d239b285d25"}
Nov 02 13:43:34.148754 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.148725    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:34.148899 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.148773    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:34.148899 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.148814    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:34.148899 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.148859    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:34.149023 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:34.148910    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle podName:d861fac3-737a-4cd7-abc6-4af75b4c1d67 nodeName:}" failed. No retries permitted until 2026-11-02 13:43:38.148886221 +0000 UTC m=+8.900190637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle") pod "router-default-5466575dd5-gsvt5" (UID: "d861fac3-737a-4cd7-abc6-4af75b4c1d67") : configmap references non-existent config key: service-ca.crt
Nov 02 13:43:34.149023 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:34.148955    5034 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:34.149023 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:34.148959    5034 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 13:43:34.149023 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:34.149000    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls podName:028f7fde-fe8a-4d88-9c6c-54193c5c490b nodeName:}" failed. No retries permitted until 2026-11-02 13:43:38.148986586 +0000 UTC m=+8.900291016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2m5tg" (UID: "028f7fde-fe8a-4d88-9c6c-54193c5c490b") : secret "insights-runtime-extractor-tls" not found
Nov 02 13:43:34.149176 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:34.149041    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs podName:91600010-0a54-4517-b9a7-1cc38b075abd nodeName:}" failed. No retries permitted until 2026-11-02 13:43:38.149025125 +0000 UTC m=+8.900329559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs") pod "network-metrics-daemon-g8jrw" (UID: "91600010-0a54-4517-b9a7-1cc38b075abd") : secret "metrics-daemon-secret" not found
Nov 02 13:43:34.150993 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.150968    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63881d4e-a3bf-4948-a369-c892baba23a8-metrics-tls\") pod \"dns-default-fzlvp\" (UID: \"63881d4e-a3bf-4948-a369-c892baba23a8\") " pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:34.311044 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.311016    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:34.490071 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.490044    5034 generic.go:356] "Generic (PLEG): container finished" podID="27e3e0a4-dc29-4e68-9344-f82ea07cb067" containerID="3782015e076b004d734332b23b2bc704dd58e57ef84816ceb3e97e4af27763ee" exitCode=0
Nov 02 13:43:34.490451 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.490122    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6" event={"ID":"27e3e0a4-dc29-4e68-9344-f82ea07cb067","Type":"ContainerDied","Data":"3782015e076b004d734332b23b2bc704dd58e57ef84816ceb3e97e4af27763ee"}
Nov 02 13:43:34.490508 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.490473    5034 scope.go:122] "RemoveContainer" containerID="3782015e076b004d734332b23b2bc704dd58e57ef84816ceb3e97e4af27763ee"
Nov 02 13:43:34.492177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.492153    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-f79wg" event={"ID":"8646ac2d-cfd2-4b7c-929a-906815e6551f","Type":"ContainerStarted","Data":"0da50c343fbf19b365fec4ea13a72fd4739a077381e9dde3db9e465610655134"}
Nov 02 13:43:34.493664 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.493646    5034 generic.go:356] "Generic (PLEG): container finished" podID="fb81289b-9a76-4529-a6f5-649ea7c85d67" containerID="5916ca1f7909bd58b5f42ce47aac38ac0f60250d29085c76b17e5b68e1bdda85" exitCode=0
Nov 02 13:43:34.493749 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.493716    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n" event={"ID":"fb81289b-9a76-4529-a6f5-649ea7c85d67","Type":"ContainerDied","Data":"5916ca1f7909bd58b5f42ce47aac38ac0f60250d29085c76b17e5b68e1bdda85"}
Nov 02 13:43:34.494260 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.494242    5034 scope.go:122] "RemoveContainer" containerID="5916ca1f7909bd58b5f42ce47aac38ac0f60250d29085c76b17e5b68e1bdda85"
Nov 02 13:43:34.609916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:34.609890    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fzlvp"]
Nov 02 13:43:34.611253 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:34.611224    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63881d4e_a3bf_4948_a369_c892baba23a8.slice/crio-3e463ef0cb4976716e983b5ed58fd200a0995d60bea3c4e83a38f65119fa5a68 WatchSource:0}: Error finding container 3e463ef0cb4976716e983b5ed58fd200a0995d60bea3c4e83a38f65119fa5a68: Status 404 returned error can't find the container with id 3e463ef0cb4976716e983b5ed58fd200a0995d60bea3c4e83a38f65119fa5a68
Nov 02 13:43:35.116180 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:35.116137    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:43:35.498523 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:35.498483    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fzlvp" event={"ID":"63881d4e-a3bf-4948-a369-c892baba23a8","Type":"ContainerStarted","Data":"3e463ef0cb4976716e983b5ed58fd200a0995d60bea3c4e83a38f65119fa5a68"}
Nov 02 13:43:35.500449 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:35.500417    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mtl2n" event={"ID":"fb81289b-9a76-4529-a6f5-649ea7c85d67","Type":"ContainerStarted","Data":"f338c3c5013365ef254fd483ca5dbeda37221b7ae9b888d1575bc3e87d1825c7"}
Nov 02 13:43:35.502342 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:35.502322    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-xkkj6" event={"ID":"27e3e0a4-dc29-4e68-9344-f82ea07cb067","Type":"ContainerStarted","Data":"7df8ba279342e8526142a2a2c5ff17d1e9237b6a8b2449065fe96085c8821fcf"}
Nov 02 13:43:36.507680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:36.507644    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fzlvp" event={"ID":"63881d4e-a3bf-4948-a369-c892baba23a8","Type":"ContainerStarted","Data":"1e8f5bd05ed6ca6be1854300fd117757f4d02691517306cc284a018b89d09ac9"}
Nov 02 13:43:36.508066 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:36.507702    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fzlvp" event={"ID":"63881d4e-a3bf-4948-a369-c892baba23a8","Type":"ContainerStarted","Data":"1c9818534be4ae9f10ed08cde540b2dfc0e01ece81ccd3611643296441dbad79"}
Nov 02 13:43:36.508066 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:36.507765    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:36.821237 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:36.821169    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:37.687852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:37.687824    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:37.688210 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:37.688123    5034 scope.go:122] "RemoveContainer" containerID="9edd8c64cb2d5f3842349b869336e15eb19b2e89ba5407353efbef31430d160e"
Nov 02 13:43:37.688275 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:37.688257    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-zfzrk_openshift-console-operator(398a3f5c-7cbe-448d-8f16-3360c18a38bf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" podUID="398a3f5c-7cbe-448d-8f16-3360c18a38bf"
Nov 02 13:43:38.181851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.181822    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:38.182000 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.181864    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:38.182000 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.181888    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:38.182402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.182378    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d861fac3-737a-4cd7-abc6-4af75b4c1d67-service-ca-bundle\") pod \"router-default-5466575dd5-gsvt5\" (UID: \"d861fac3-737a-4cd7-abc6-4af75b4c1d67\") " pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:38.184145 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.184125    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91600010-0a54-4517-b9a7-1cc38b075abd-metrics-certs\") pod \"network-metrics-daemon-g8jrw\" (UID: \"91600010-0a54-4517-b9a7-1cc38b075abd\") " pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:38.184211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.184128    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/028f7fde-fe8a-4d88-9c6c-54193c5c490b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2m5tg\" (UID: \"028f7fde-fe8a-4d88-9c6c-54193c5c490b\") " pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:38.199067 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.199048    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:38.209635 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.209490    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8jrw"
Nov 02 13:43:38.212619 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.212598    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2m5tg"
Nov 02 13:43:38.526872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.526838    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g8jrw"]
Nov 02 13:43:38.528659 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:38.528632    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91600010_0a54_4517_b9a7_1cc38b075abd.slice/crio-eab527f9256c9a6ec120a24e0b9648b1612ff05ed811ebd6dc7e91f23a4f6a62 WatchSource:0}: Error finding container eab527f9256c9a6ec120a24e0b9648b1612ff05ed811ebd6dc7e91f23a4f6a62: Status 404 returned error can't find the container with id eab527f9256c9a6ec120a24e0b9648b1612ff05ed811ebd6dc7e91f23a4f6a62
Nov 02 13:43:38.529995 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.529970    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2m5tg"]
Nov 02 13:43:38.530404 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:38.530379    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028f7fde_fe8a_4d88_9c6c_54193c5c490b.slice/crio-8024a9c7c1d70a331d55b4b44c7a5e8e8404838252dbda52c55b7d8ccfa46f9d WatchSource:0}: Error finding container 8024a9c7c1d70a331d55b4b44c7a5e8e8404838252dbda52c55b7d8ccfa46f9d: Status 404 returned error can't find the container with id 8024a9c7c1d70a331d55b4b44c7a5e8e8404838252dbda52c55b7d8ccfa46f9d
Nov 02 13:43:38.533879 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:38.533859    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5466575dd5-gsvt5"]
Nov 02 13:43:38.539092 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:38.539066    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd861fac3_737a_4cd7_abc6_4af75b4c1d67.slice/crio-12c97de12503016781408fcddc1619520eae89baf3e26b23864627ab74f8d985 WatchSource:0}: Error finding container 12c97de12503016781408fcddc1619520eae89baf3e26b23864627ab74f8d985: Status 404 returned error can't find the container with id 12c97de12503016781408fcddc1619520eae89baf3e26b23864627ab74f8d985
Nov 02 13:43:39.346716 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:39.346689    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:39.347069 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:39.346958    5034 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 13:43:39.520794 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:39.520746    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5466575dd5-gsvt5" event={"ID":"d861fac3-737a-4cd7-abc6-4af75b4c1d67","Type":"ContainerStarted","Data":"74484cc09676d5fc2f3bf66debdec5c30d4c351dbac4e195138e653f3c7aea58"}
Nov 02 13:43:39.520894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:39.520830    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5466575dd5-gsvt5" event={"ID":"d861fac3-737a-4cd7-abc6-4af75b4c1d67","Type":"ContainerStarted","Data":"12c97de12503016781408fcddc1619520eae89baf3e26b23864627ab74f8d985"}
Nov 02 13:43:39.523557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:39.523528    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2m5tg" event={"ID":"028f7fde-fe8a-4d88-9c6c-54193c5c490b","Type":"ContainerStarted","Data":"38944415a92b20bb59c0924a3a49711c5f7e2214bf9ec14e634219848a0c7829"}
Nov 02 13:43:39.523702 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:39.523582    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2m5tg" event={"ID":"028f7fde-fe8a-4d88-9c6c-54193c5c490b","Type":"ContainerStarted","Data":"2741595e78c7e462e7b0ede8bdfe0731991cc57bad1dc7fbeb0e00fc43254cdd"}
Nov 02 13:43:39.523702 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:39.523604    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2m5tg" event={"ID":"028f7fde-fe8a-4d88-9c6c-54193c5c490b","Type":"ContainerStarted","Data":"8024a9c7c1d70a331d55b4b44c7a5e8e8404838252dbda52c55b7d8ccfa46f9d"}
Nov 02 13:43:39.525279 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:39.525223    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g8jrw" event={"ID":"91600010-0a54-4517-b9a7-1cc38b075abd","Type":"ContainerStarted","Data":"eab527f9256c9a6ec120a24e0b9648b1612ff05ed811ebd6dc7e91f23a4f6a62"}
Nov 02 13:43:40.199274 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:40.199244    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:40.201799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:40.201758    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:40.532650 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:40.532608    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g8jrw" event={"ID":"91600010-0a54-4517-b9a7-1cc38b075abd","Type":"ContainerStarted","Data":"8eab17457e3a07625d7bac67b914435cabeb4bb3f404abc1a69392f3a82024d5"}
Nov 02 13:43:40.533152 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:40.532701    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g8jrw" event={"ID":"91600010-0a54-4517-b9a7-1cc38b075abd","Type":"ContainerStarted","Data":"a3e96abe64ea6bdfa8f2f7cc5f3666da067d8cac630f690b7ac9213fe3ea38c7"}
Nov 02 13:43:41.264532 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:41.264497    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:41.265752 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:41.265727    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5466575dd5-gsvt5"
Nov 02 13:43:41.538932 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:41.538850    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2m5tg" event={"ID":"028f7fde-fe8a-4d88-9c6c-54193c5c490b","Type":"ContainerStarted","Data":"d7a6c81702c73cd19791249f30edad1b1d808506c30130552b725aa7effb6649"}
Nov 02 13:43:42.386692 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:42.386664    5034 scope.go:122] "RemoveContainer" containerID="9edd8c64cb2d5f3842349b869336e15eb19b2e89ba5407353efbef31430d160e"
Nov 02 13:43:42.543922 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:42.543897    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 13:43:42.544297 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:42.544010    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk" event={"ID":"398a3f5c-7cbe-448d-8f16-3360c18a38bf","Type":"ContainerStarted","Data":"d56bb4167789141350cf450aefc4e855f4974c9ec7fed7d0504d918bfb262c74"}
Nov 02 13:43:44.763040 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:44.763004    5034 patch_prober.go:29] interesting pod/image-registry-8c6787bcd-psbdk container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 13:43:44.763408 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:44.763063    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk" podUID="633c4858-9dc5-4049-aec3-8a14171652ae" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:43:46.513614 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:46.513589    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fzlvp"
Nov 02 13:43:46.825382 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:46.825313    5034 patch_prober.go:29] interesting pod/image-registry-8c6787bcd-psbdk container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 13:43:46.825382 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:46.825355    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk" podUID="633c4858-9dc5-4049-aec3-8a14171652ae" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:43:47.653076 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:47.653051    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rfknw"
Nov 02 13:43:48.051097 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:48.051068    5034 patch_prober.go:29] interesting pod/image-registry-5d555fdbf5-mnstt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 13:43:48.051257 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:48.051123    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt" podUID="3fa80f89-f24d-4f86-940c-864dbf177a5c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:43:48.411504 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:48.411427    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:43:49.719048 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.719006    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-66rdf-vjshb"]
Nov 02 13:43:49.723623 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.723599    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.735626 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.735598    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d555fdbf5-mnstt"]
Nov 02 13:43:49.741530 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.741502    5034 patch_prober.go:29] interesting pod/image-registry-5d555fdbf5-mnstt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 02 13:43:49.741653 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.741545    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt" podUID="3fa80f89-f24d-4f86-940c-864dbf177a5c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:43:49.761623 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.761596    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-66rdf-vjshb"]
Nov 02 13:43:49.785709 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.785683    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-5tvk9"]
Nov 02 13:43:49.789021 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.788993    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:43:49.796835 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.796811    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"]
Nov 02 13:43:49.800158 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.800139    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-858dfd774b-4m89s"]
Nov 02 13:43:49.800276 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.800262    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"
Nov 02 13:43:49.803539 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.803521    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:49.817927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.817906    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 13:43:49.818030 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.817945    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 13:43:49.818030 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.817992    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-grkzz\""
Nov 02 13:43:49.826755 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.826729    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858dfd774b-4m89s"]
Nov 02 13:43:49.840491 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.840467    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5tvk9"]
Nov 02 13:43:49.841648 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.841624    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"]
Nov 02 13:43:49.842417 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.842397    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7mx2k"
Nov 02 13:43:49.853605 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.853471    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 02 13:43:49.854651 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.854631    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 02 13:43:49.854763 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.854693    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 02 13:43:49.868383 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.868358    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57cccf57dc-xn8xj"]
Nov 02 13:43:49.871644 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.871624    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:49.880011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.879987    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm95z\" (UniqueName: \"kubernetes.io/projected/cfa38906-422d-4391-bfc0-a696866c7a20-kube-api-access-dm95z\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.880097 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.880028    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-service-ca-bundle\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.880097 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.880060    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d0dbccc-3b95-4872-988a-b8df4e6b02f0-tmp\") pod \"downloads-6dd789c8b7-5tvk9\" (UID: \"1d0dbccc-3b95-4872-988a-b8df4e6b02f0\") " pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:43:49.880188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.880172    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqkmx\" (UniqueName: \"kubernetes.io/projected/1d0dbccc-3b95-4872-988a-b8df4e6b02f0-kube-api-access-wqkmx\") pod \"downloads-6dd789c8b7-5tvk9\" (UID: \"1d0dbccc-3b95-4872-988a-b8df4e6b02f0\") " pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:43:49.880220 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.880197    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cfa38906-422d-4391-bfc0-a696866c7a20-archives-path\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.880220 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.880216    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-trusted-ca-bundle\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.905873 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.905851    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57cccf57dc-xn8xj"]
Nov 02 13:43:49.916184 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.916156    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 02 13:43:49.916340 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.916319    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 02 13:43:49.916411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.916339    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6x5k2\""
Nov 02 13:43:49.916411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.916319    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 13:43:49.916411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.916363    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tmfcw\""
Nov 02 13:43:49.980809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.980734    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-serving-cert\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:49.980809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.980786    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aae1ab7a-b2c1-4716-a1af-82815b203327-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rmw2p\" (UID: \"aae1ab7a-b2c1-4716-a1af-82815b203327\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"
Nov 02 13:43:49.980985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.980853    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqkmx\" (UniqueName: \"kubernetes.io/projected/1d0dbccc-3b95-4872-988a-b8df4e6b02f0-kube-api-access-wqkmx\") pod \"downloads-6dd789c8b7-5tvk9\" (UID: \"1d0dbccc-3b95-4872-988a-b8df4e6b02f0\") " pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:43:49.980985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.980882    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cfa38906-422d-4391-bfc0-a696866c7a20-archives-path\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.980985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.980904    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-trusted-ca-bundle\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.980985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.980932    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgjsv\" (UniqueName: \"kubernetes.io/projected/7a0f79c9-79d9-41da-9617-ef20d42df29d-kube-api-access-jgjsv\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:49.980985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.980962    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-service-ca\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:49.981204 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.980983    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a0f79c9-79d9-41da-9617-ef20d42df29d-image-registry-private-configuration\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:49.981204 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981141    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a0f79c9-79d9-41da-9617-ef20d42df29d-registry-tls\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:49.981204 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981197    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm95z\" (UniqueName: \"kubernetes.io/projected/cfa38906-422d-4391-bfc0-a696866c7a20-kube-api-access-dm95z\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.981320 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981224    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-oauth-serving-cert\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:49.981320 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981271    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cfa38906-422d-4391-bfc0-a696866c7a20-archives-path\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.981320 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981301    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a0f79c9-79d9-41da-9617-ef20d42df29d-registry-certificates\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:49.981414 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981345    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a0f79c9-79d9-41da-9617-ef20d42df29d-trusted-ca\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:49.981414 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981406    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a0f79c9-79d9-41da-9617-ef20d42df29d-ca-trust-extracted\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:49.981476 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981437    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a0f79c9-79d9-41da-9617-ef20d42df29d-bound-sa-token\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:49.981514 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981482    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-service-ca-bundle\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.981558 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981518    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-tmp\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:49.981558 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981535    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-oauth-config\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:49.981558 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981552    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a0f79c9-79d9-41da-9617-ef20d42df29d-installation-pull-secrets\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:49.981766 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981745    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtjt\" (UniqueName: \"kubernetes.io/projected/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-kube-api-access-9mtjt\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:49.981852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981828    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-config\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:49.981902 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.981881    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d0dbccc-3b95-4872-988a-b8df4e6b02f0-tmp\") pod \"downloads-6dd789c8b7-5tvk9\" (UID: \"1d0dbccc-3b95-4872-988a-b8df4e6b02f0\") " pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:43:49.982035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.982018    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-service-ca-bundle\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.982104 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.982088    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-trusted-ca-bundle\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:49.982249 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:49.982230    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d0dbccc-3b95-4872-988a-b8df4e6b02f0-tmp\") pod \"downloads-6dd789c8b7-5tvk9\" (UID: \"1d0dbccc-3b95-4872-988a-b8df4e6b02f0\") " pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:43:50.020139 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.020115    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm95z\" (UniqueName: \"kubernetes.io/projected/cfa38906-422d-4391-bfc0-a696866c7a20-kube-api-access-dm95z\") pod \"periodic-gathering-66rdf-vjshb\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") " pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:50.020305 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.020288    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqkmx\" (UniqueName: \"kubernetes.io/projected/1d0dbccc-3b95-4872-988a-b8df4e6b02f0-kube-api-access-wqkmx\") pod \"downloads-6dd789c8b7-5tvk9\" (UID: \"1d0dbccc-3b95-4872-988a-b8df4e6b02f0\") " pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:43:50.039974 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.039951    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:43:50.082482 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082452    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjsv\" (UniqueName: \"kubernetes.io/projected/7a0f79c9-79d9-41da-9617-ef20d42df29d-kube-api-access-jgjsv\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.082482 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082486    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-service-ca\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.082666 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082505    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a0f79c9-79d9-41da-9617-ef20d42df29d-image-registry-private-configuration\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.082666 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082538    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a0f79c9-79d9-41da-9617-ef20d42df29d-registry-tls\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.082666 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082571    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-oauth-serving-cert\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.082666 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082595    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a0f79c9-79d9-41da-9617-ef20d42df29d-registry-certificates\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.082666 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082621    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a0f79c9-79d9-41da-9617-ef20d42df29d-trusted-ca\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.082666 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082655    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a0f79c9-79d9-41da-9617-ef20d42df29d-ca-trust-extracted\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.082958 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082684    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a0f79c9-79d9-41da-9617-ef20d42df29d-bound-sa-token\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.082958 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082715    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-tmp\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.082958 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082740    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-oauth-config\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.082958 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.082936    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a0f79c9-79d9-41da-9617-ef20d42df29d-installation-pull-secrets\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.083583 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.083556    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtjt\" (UniqueName: \"kubernetes.io/projected/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-kube-api-access-9mtjt\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.083803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.083758    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-service-ca\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.083885 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.083609    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-config\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.083952 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.083884    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-serving-cert\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.084130 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.083929    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aae1ab7a-b2c1-4716-a1af-82815b203327-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rmw2p\" (UID: \"aae1ab7a-b2c1-4716-a1af-82815b203327\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"
Nov 02 13:43:50.084416 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.083484    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7a0f79c9-79d9-41da-9617-ef20d42df29d-ca-trust-extracted\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.084504 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.083561    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-tmp\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.084560 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.084538    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7a0f79c9-79d9-41da-9617-ef20d42df29d-registry-certificates\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.084614 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.084576    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-config\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.085468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.085443    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-oauth-serving-cert\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.085912 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.085886    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7a0f79c9-79d9-41da-9617-ef20d42df29d-image-registry-private-configuration\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.088503 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.088464    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7a0f79c9-79d9-41da-9617-ef20d42df29d-trusted-ca\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.088627 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.088579    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-oauth-config\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.089957 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.089047    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7a0f79c9-79d9-41da-9617-ef20d42df29d-installation-pull-secrets\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.104228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.104202    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:43:50.105361 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.105333    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/aae1ab7a-b2c1-4716-a1af-82815b203327-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rmw2p\" (UID: \"aae1ab7a-b2c1-4716-a1af-82815b203327\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"
Nov 02 13:43:50.106508 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.106487    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7a0f79c9-79d9-41da-9617-ef20d42df29d-registry-tls\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.107881 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.107855    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-serving-cert\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.111032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.110986    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgjsv\" (UniqueName: \"kubernetes.io/projected/7a0f79c9-79d9-41da-9617-ef20d42df29d-kube-api-access-jgjsv\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.111432 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.111408    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtjt\" (UniqueName: \"kubernetes.io/projected/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-kube-api-access-9mtjt\") pod \"console-858dfd774b-4m89s\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") " pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.115584 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.115474    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7a0f79c9-79d9-41da-9617-ef20d42df29d-bound-sa-token\") pod \"image-registry-57cccf57dc-xn8xj\" (UID: \"7a0f79c9-79d9-41da-9617-ef20d42df29d\") " pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.118945 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.118918    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"
Nov 02 13:43:50.119228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.119144    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:43:50.189794 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.185875    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:50.354141 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.354082    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-66rdf-vjshb"]
Nov 02 13:43:50.355112 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:50.355087    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa38906_422d_4391_bfc0_a696866c7a20.slice/crio-35756f88dc70196277f3aa7d5c82f6500d63bdbc31015bbff876b20e87e186a5 WatchSource:0}: Error finding container 35756f88dc70196277f3aa7d5c82f6500d63bdbc31015bbff876b20e87e186a5: Status 404 returned error can't find the container with id 35756f88dc70196277f3aa7d5c82f6500d63bdbc31015bbff876b20e87e186a5
Nov 02 13:43:50.430826 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.430802    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5tvk9"]
Nov 02 13:43:50.432579 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:50.432553    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0dbccc_3b95_4872_988a_b8df4e6b02f0.slice/crio-fb58a755558b883ec1a1a91dc3013ca99ec3462b3c866c35863175a4b88f3d2d WatchSource:0}: Error finding container fb58a755558b883ec1a1a91dc3013ca99ec3462b3c866c35863175a4b88f3d2d: Status 404 returned error can't find the container with id fb58a755558b883ec1a1a91dc3013ca99ec3462b3c866c35863175a4b88f3d2d
Nov 02 13:43:50.463515 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.463475    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"]
Nov 02 13:43:50.463619 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:50.463482    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133f2f2b_96a4_4bc3_a844_9f19a068dbf7.slice/crio-0b91b3146d34bc6efd63e62223aaaf27d1c46c467d375c75cf3d454751612a1f WatchSource:0}: Error finding container 0b91b3146d34bc6efd63e62223aaaf27d1c46c467d375c75cf3d454751612a1f: Status 404 returned error can't find the container with id 0b91b3146d34bc6efd63e62223aaaf27d1c46c467d375c75cf3d454751612a1f
Nov 02 13:43:50.464069 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:50.464041    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae1ab7a_b2c1_4716_a1af_82815b203327.slice/crio-c3a912ced1229fe8bf8b6d8d15026637cc0a5360a5ba0b6245fd42c0c7dd8bfa WatchSource:0}: Error finding container c3a912ced1229fe8bf8b6d8d15026637cc0a5360a5ba0b6245fd42c0c7dd8bfa: Status 404 returned error can't find the container with id c3a912ced1229fe8bf8b6d8d15026637cc0a5360a5ba0b6245fd42c0c7dd8bfa
Nov 02 13:43:50.464907 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.464876    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858dfd774b-4m89s"]
Nov 02 13:43:50.524208 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.524184    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57cccf57dc-xn8xj"]
Nov 02 13:43:50.525012 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:50.524989    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0f79c9_79d9_41da_9617_ef20d42df29d.slice/crio-022e7308a7f4c5c70da7cc19da4b7d5dbbf0631063b35d553c9e3ebe586829d1 WatchSource:0}: Error finding container 022e7308a7f4c5c70da7cc19da4b7d5dbbf0631063b35d553c9e3ebe586829d1: Status 404 returned error can't find the container with id 022e7308a7f4c5c70da7cc19da4b7d5dbbf0631063b35d553c9e3ebe586829d1
Nov 02 13:43:50.577464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.577434    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p" event={"ID":"aae1ab7a-b2c1-4716-a1af-82815b203327","Type":"ContainerStarted","Data":"c3a912ced1229fe8bf8b6d8d15026637cc0a5360a5ba0b6245fd42c0c7dd8bfa"}
Nov 02 13:43:50.578846 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.578820    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5tvk9" event={"ID":"1d0dbccc-3b95-4872-988a-b8df4e6b02f0","Type":"ContainerStarted","Data":"fb58a755558b883ec1a1a91dc3013ca99ec3462b3c866c35863175a4b88f3d2d"}
Nov 02 13:43:50.580645 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.580622    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj" event={"ID":"7a0f79c9-79d9-41da-9617-ef20d42df29d","Type":"ContainerStarted","Data":"022e7308a7f4c5c70da7cc19da4b7d5dbbf0631063b35d553c9e3ebe586829d1"}
Nov 02 13:43:50.582188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.582159    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858dfd774b-4m89s" event={"ID":"133f2f2b-96a4-4bc3-a844-9f19a068dbf7","Type":"ContainerStarted","Data":"0b91b3146d34bc6efd63e62223aaaf27d1c46c467d375c75cf3d454751612a1f"}
Nov 02 13:43:50.584032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.583987    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-66rdf-vjshb" event={"ID":"cfa38906-422d-4391-bfc0-a696866c7a20","Type":"ContainerStarted","Data":"22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa"}
Nov 02 13:43:50.584130 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.584074    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-66rdf-vjshb" event={"ID":"cfa38906-422d-4391-bfc0-a696866c7a20","Type":"ContainerStarted","Data":"35756f88dc70196277f3aa7d5c82f6500d63bdbc31015bbff876b20e87e186a5"}
Nov 02 13:43:50.634294 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:50.634226    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-66rdf-vjshb" podStartSLOduration=1.634210234 podStartE2EDuration="1.634210234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:50.632956268 +0000 UTC m=+21.384260705" watchObservedRunningTime="2026-11-02 13:43:50.634210234 +0000 UTC m=+21.385514666"
Nov 02 13:43:51.592524 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:51.592450    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj" event={"ID":"7a0f79c9-79d9-41da-9617-ef20d42df29d","Type":"ContainerStarted","Data":"2b083ccb877158a9005b3d663ad5fe4e27a7386a90152f455c275c837b10e74d"}
Nov 02 13:43:51.593051 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:51.592884    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:43:51.819189 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:51.819162    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fzlvp_63881d4e-a3bf-4948-a369-c892baba23a8/dns/0.log"
Nov 02 13:43:52.018396 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.018342    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fzlvp_63881d4e-a3bf-4948-a369-c892baba23a8/kube-rbac-proxy/0.log"
Nov 02 13:43:52.544664 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.544614    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:52.549269 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.549247    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-zfzrk"
Nov 02 13:43:52.600420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.599937    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p" event={"ID":"aae1ab7a-b2c1-4716-a1af-82815b203327","Type":"ContainerStarted","Data":"ea3bb89e2769eae02c11acd44540aef9094a74d8fbfea9dbb00c6791e6d07d1b"}
Nov 02 13:43:52.600420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.600253    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"
Nov 02 13:43:52.606859 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.606833    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p"
Nov 02 13:43:52.617736 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.617691    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj" podStartSLOduration=3.617672831 podStartE2EDuration="3.617672831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:51.671112906 +0000 UTC m=+22.422417344" watchObservedRunningTime="2026-11-02 13:43:52.617672831 +0000 UTC m=+23.368977270"
Nov 02 13:43:52.633004 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.632917    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t6vhd_26f2db88-b241-451f-8f1e-9f8c68ad0e73/dns-node-resolver/0.log"
Nov 02 13:43:52.668323 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.668041    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rmw2p" podStartSLOduration=2.294103899 podStartE2EDuration="3.668022062s" totalImagesPullingTime="1.373918163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:52.667042526 +0000 UTC m=+23.418346959" watchObservedRunningTime="2026-11-02 13:43:52.668022062 +0000 UTC m=+23.419326503"
Nov 02 13:43:52.819230 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:52.819157    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57cccf57dc-xn8xj_7a0f79c9-79d9-41da-9617-ef20d42df29d/registry/0.log"
Nov 02 13:43:53.018244 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:53.018216    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d555fdbf5-mnstt_3fa80f89-f24d-4f86-940c-864dbf177a5c/registry/0.log"
Nov 02 13:43:53.219562 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:53.219502    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8c6787bcd-psbdk_633c4858-9dc5-4049-aec3-8a14171652ae/registry/0.log"
Nov 02 13:43:53.418233 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:53.418204    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-29jl9_c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd/node-ca/0.log"
Nov 02 13:43:54.019481 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:54.019359    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5466575dd5-gsvt5_d861fac3-737a-4cd7-abc6-4af75b4c1d67/router/0.log"
Nov 02 13:43:54.220465 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:54.220431    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b7nn8_df16fea1-6bca-47f3-b9bd-35716e3d50f3/serve-healthcheck-canary/0.log"
Nov 02 13:43:54.612408 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:54.612321    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858dfd774b-4m89s" event={"ID":"133f2f2b-96a4-4bc3-a844-9f19a068dbf7","Type":"ContainerStarted","Data":"9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021"}
Nov 02 13:43:54.668613 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:54.668563    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-858dfd774b-4m89s" podStartSLOduration=1.911328502 podStartE2EDuration="5.668545417s" totalImagesPullingTime="3.757216915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:43:54.6682629 +0000 UTC m=+25.419567340" watchObservedRunningTime="2026-11-02 13:43:54.668545417 +0000 UTC m=+25.419849856"
Nov 02 13:43:56.826396 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:56.826368    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:43:58.020864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.020827    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-btv25"]
Nov 02 13:43:58.059714 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.059686    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.062533 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.062501    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.062668 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.062542    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.062668 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.062654    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-tls\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.062803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.062694    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9afd0929-6d3a-418c-bd2e-902443e20a7a-metrics-client-ca\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.062803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.062748    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49xjr\" (UniqueName: \"kubernetes.io/projected/9afd0929-6d3a-418c-bd2e-902443e20a7a-kube-api-access-49xjr\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.062906 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.062832    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9afd0929-6d3a-418c-bd2e-902443e20a7a-root\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.062906 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.062861    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-textfile\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.063003 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.062909    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-wtmp\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.063003 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.062930    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9afd0929-6d3a-418c-bd2e-902443e20a7a-sys\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.085452 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.085423    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 13:43:58.085452 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.085434    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 13:43:58.085629 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.085522    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hnbmg\""
Nov 02 13:43:58.102134 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.102107    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 13:43:58.102308 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.102290    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 13:43:58.164030 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164003    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-wtmp\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164030 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164040    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9afd0929-6d3a-418c-bd2e-902443e20a7a-sys\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164250 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164068    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164250 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164139    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9afd0929-6d3a-418c-bd2e-902443e20a7a-sys\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164250 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164189    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164250 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164243    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-wtmp\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164434 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164266    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-tls\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164434 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164319    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9afd0929-6d3a-418c-bd2e-902443e20a7a-metrics-client-ca\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164434 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164374    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49xjr\" (UniqueName: \"kubernetes.io/projected/9afd0929-6d3a-418c-bd2e-902443e20a7a-kube-api-access-49xjr\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164434 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164402    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9afd0929-6d3a-418c-bd2e-902443e20a7a-root\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164434 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164426    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-textfile\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164597 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:58.164440    5034 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Nov 02 13:43:58.164597 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:43:58.164513    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-tls podName:9afd0929-6d3a-418c-bd2e-902443e20a7a nodeName:}" failed. No retries permitted until 2026-11-02 13:43:58.664491379 +0000 UTC m=+29.415795798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-tls") pod "node-exporter-btv25" (UID: "9afd0929-6d3a-418c-bd2e-902443e20a7a") : secret "node-exporter-tls" not found
Nov 02 13:43:58.164770 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164751    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164860 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164756    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-textfile\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164860 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164839    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9afd0929-6d3a-418c-bd2e-902443e20a7a-root\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.164951 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.164926    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9afd0929-6d3a-418c-bd2e-902443e20a7a-metrics-client-ca\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.166535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.166512    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.211182 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.211155    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49xjr\" (UniqueName: \"kubernetes.io/projected/9afd0929-6d3a-418c-bd2e-902443e20a7a-kube-api-access-49xjr\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.667455 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.667419    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-tls\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.669921 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.669898    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9afd0929-6d3a-418c-bd2e-902443e20a7a-node-exporter-tls\") pod \"node-exporter-btv25\" (UID: \"9afd0929-6d3a-418c-bd2e-902443e20a7a\") " pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.730831 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:58.730803    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-btv25"
Nov 02 13:43:58.774046 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:43:58.774013    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afd0929_6d3a_418c_bd2e_902443e20a7a.slice/crio-4e78d329e5a595d9de2ebe061cfb98b5f8db2387d77be0a7caa43fa2d2655a5f WatchSource:0}: Error finding container 4e78d329e5a595d9de2ebe061cfb98b5f8db2387d77be0a7caa43fa2d2655a5f: Status 404 returned error can't find the container with id 4e78d329e5a595d9de2ebe061cfb98b5f8db2387d77be0a7caa43fa2d2655a5f
Nov 02 13:43:59.639915 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:59.639872    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-btv25" event={"ID":"9afd0929-6d3a-418c-bd2e-902443e20a7a","Type":"ContainerStarted","Data":"4e78d329e5a595d9de2ebe061cfb98b5f8db2387d77be0a7caa43fa2d2655a5f"}
Nov 02 13:43:59.744526 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:43:59.744490    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:44:00.119335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:00.119298    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:44:00.119536 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:00.119366    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:44:00.125175 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:00.125146    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:44:00.648109 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:00.648068    5034 generic.go:356] "Generic (PLEG): container finished" podID="9afd0929-6d3a-418c-bd2e-902443e20a7a" containerID="6e3aefc33bf3d2f8e4204dd81b5e083a00dc5336decc67760fba70f6fbbd1d3f" exitCode=0
Nov 02 13:44:00.648516 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:00.648163    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-btv25" event={"ID":"9afd0929-6d3a-418c-bd2e-902443e20a7a","Type":"ContainerDied","Data":"6e3aefc33bf3d2f8e4204dd81b5e083a00dc5336decc67760fba70f6fbbd1d3f"}
Nov 02 13:44:00.652878 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:00.652855    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:44:01.659435 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:01.659397    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-btv25" event={"ID":"9afd0929-6d3a-418c-bd2e-902443e20a7a","Type":"ContainerStarted","Data":"845b8a984493d6112a6a20e38a5573b5930bf7416e15b8fba840d84ed95dff4e"}
Nov 02 13:44:01.659894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:01.659467    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-btv25" event={"ID":"9afd0929-6d3a-418c-bd2e-902443e20a7a","Type":"ContainerStarted","Data":"f671ea33a15689d965f4f6e5145110af3115828b126e5908bf840e8575806dcb"}
Nov 02 13:44:01.723093 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:01.723024    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-btv25" podStartSLOduration=3.223704814 podStartE2EDuration="4.723009655s" totalImagesPullingTime="1.402779367s" totalInitContainerRuntime="96.525474ms" isStatefulPod=true podCreationTimestamp="2026-11-02 13:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:01.722361091 +0000 UTC m=+32.473665528" watchObservedRunningTime="2026-11-02 13:44:01.723009655 +0000 UTC m=+32.474314151"
Nov 02 13:44:02.193735 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.193688    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5456f4ccf6-gsh5j"]
Nov 02 13:44:02.217968 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.217939    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.224565 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.224529    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5456f4ccf6-gsh5j"]
Nov 02 13:44:02.276674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.276642    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 02 13:44:02.300978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.300948    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-trusted-ca-bundle\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.301140 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.301004    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-service-ca\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.301140 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.301129    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9cbabf2-f039-4466-97dc-601c191e8d9b-tmp\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.301228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.301165    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-oauth-serving-cert\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.301274 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.301219    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4w62\" (UniqueName: \"kubernetes.io/projected/f9cbabf2-f039-4466-97dc-601c191e8d9b-kube-api-access-p4w62\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.301274 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.301256    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-oauth-config\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.301338 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.301291    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-config\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.301338 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.301323    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-serving-cert\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402024 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.401988    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-config\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402202 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.402030    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-serving-cert\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402202 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.402101    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-trusted-ca-bundle\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.402253    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-service-ca\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.402300    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9cbabf2-f039-4466-97dc-601c191e8d9b-tmp\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402428 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.402328    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-oauth-serving-cert\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402428 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.402359    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4w62\" (UniqueName: \"kubernetes.io/projected/f9cbabf2-f039-4466-97dc-601c191e8d9b-kube-api-access-p4w62\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402428 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.402398    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-oauth-config\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402733 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.402673    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9cbabf2-f039-4466-97dc-601c191e8d9b-tmp\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.402879 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.402842    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-config\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.403054 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.403028    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-trusted-ca-bundle\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.403149 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.403112    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-oauth-serving-cert\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.403245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.403221    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-service-ca\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.405116 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.405096    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-oauth-config\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.405438 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.405418    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-serving-cert\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.428021 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.427988    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4w62\" (UniqueName: \"kubernetes.io/projected/f9cbabf2-f039-4466-97dc-601c191e8d9b-kube-api-access-p4w62\") pod \"console-5456f4ccf6-gsh5j\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") " pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.549947 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.549916    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:02.896956 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:02.896928    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5456f4ccf6-gsh5j"]
Nov 02 13:44:02.898295 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:44:02.898267    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cbabf2_f039_4466_97dc_601c191e8d9b.slice/crio-9e4a952c6329267437b5cd0d52f9dc8adbe118f20874d5fcb2c1ca0aed6193f0 WatchSource:0}: Error finding container 9e4a952c6329267437b5cd0d52f9dc8adbe118f20874d5fcb2c1ca0aed6193f0: Status 404 returned error can't find the container with id 9e4a952c6329267437b5cd0d52f9dc8adbe118f20874d5fcb2c1ca0aed6193f0
Nov 02 13:44:03.670533 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:03.670497    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5456f4ccf6-gsh5j" event={"ID":"f9cbabf2-f039-4466-97dc-601c191e8d9b","Type":"ContainerStarted","Data":"d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383"}
Nov 02 13:44:03.670717 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:03.670555    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5456f4ccf6-gsh5j" event={"ID":"f9cbabf2-f039-4466-97dc-601c191e8d9b","Type":"ContainerStarted","Data":"9e4a952c6329267437b5cd0d52f9dc8adbe118f20874d5fcb2c1ca0aed6193f0"}
Nov 02 13:44:03.716815 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:03.716742    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5456f4ccf6-gsh5j" podStartSLOduration=1.716724017 podStartE2EDuration="1.716724017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:03.715471697 +0000 UTC m=+34.466776124" watchObservedRunningTime="2026-11-02 13:44:03.716724017 +0000 UTC m=+34.468028457"
Nov 02 13:44:05.119286 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:05.119255    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2gjsj"
Nov 02 13:44:07.696028 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:07.695990    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5tvk9" event={"ID":"1d0dbccc-3b95-4872-988a-b8df4e6b02f0","Type":"ContainerStarted","Data":"f4c8e6d381fa8fc787dfabaaedf6f9b56c17172403b25dc9cd65c35cd468c70d"}
Nov 02 13:44:07.696659 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:07.696630    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:44:07.718275 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:07.718236    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-5tvk9"
Nov 02 13:44:07.767432 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:07.767371    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-5tvk9" podStartSLOduration=2.363290787 podStartE2EDuration="18.767352911s" totalImagesPullingTime="16.404062124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:07.75338298 +0000 UTC m=+38.504687430" watchObservedRunningTime="2026-11-02 13:44:07.767352911 +0000 UTC m=+38.518657368"
Nov 02 13:44:12.178632 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.178546    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5456f4ccf6-gsh5j"]
Nov 02 13:44:12.243380 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.243341    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-cfc8c476c-6jvqg"]
Nov 02 13:44:12.280408 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.280368    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.284546 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.284515    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cfc8c476c-6jvqg"]
Nov 02 13:44:12.293985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.292982    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-oauth-config\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.293985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.293027    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-oauth-serving-cert\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.293985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.293060    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-service-ca\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.293985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.293086    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0c82bc5-c7ba-4225-b473-827155cd5358-tmp\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.293985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.293109    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2t2r\" (UniqueName: \"kubernetes.io/projected/a0c82bc5-c7ba-4225-b473-827155cd5358-kube-api-access-f2t2r\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.293985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.293191    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-trusted-ca-bundle\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.293985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.293230    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-serving-cert\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.293985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.293267    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-console-config\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.396478 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.396436    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-trusted-ca-bundle\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.396680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.396499    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-serving-cert\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.396680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.396535    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-console-config\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.396680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.396580    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-oauth-config\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.396680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.396617    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-oauth-serving-cert\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.396680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.396646    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-service-ca\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.396680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.396671    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0c82bc5-c7ba-4225-b473-827155cd5358-tmp\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.396959 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.396694    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2t2r\" (UniqueName: \"kubernetes.io/projected/a0c82bc5-c7ba-4225-b473-827155cd5358-kube-api-access-f2t2r\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.410808 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.397722    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-oauth-serving-cert\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.410808 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.397996    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0c82bc5-c7ba-4225-b473-827155cd5358-tmp\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.410808 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.398234    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-service-ca\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.410808 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.398554    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-trusted-ca-bundle\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.410808 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.398752    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-console-config\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.410808 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.399845    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-serving-cert\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.410808 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.399928    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-oauth-config\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.421099 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.421069    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2t2r\" (UniqueName: \"kubernetes.io/projected/a0c82bc5-c7ba-4225-b473-827155cd5358-kube-api-access-f2t2r\") pod \"console-cfc8c476c-6jvqg\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") " pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:12.551212 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.551177    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:12.627171 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:12.626949    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:13.015380 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:13.015351    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cfc8c476c-6jvqg"]
Nov 02 13:44:13.015531 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:44:13.015488    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c82bc5_c7ba_4225_b473_827155cd5358.slice/crio-6431593da6e9c85eca6a0d617b4c598908c19e094152f6444c7880a01043bb01 WatchSource:0}: Error finding container 6431593da6e9c85eca6a0d617b4c598908c19e094152f6444c7880a01043bb01: Status 404 returned error can't find the container with id 6431593da6e9c85eca6a0d617b4c598908c19e094152f6444c7880a01043bb01
Nov 02 13:44:13.610690 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:13.610663    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57cccf57dc-xn8xj"
Nov 02 13:44:13.715881 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:13.713356    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8c6787bcd-psbdk"]
Nov 02 13:44:13.755373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:13.755333    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfc8c476c-6jvqg" event={"ID":"a0c82bc5-c7ba-4225-b473-827155cd5358","Type":"ContainerStarted","Data":"6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3"}
Nov 02 13:44:13.756102 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:13.756060    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfc8c476c-6jvqg" event={"ID":"a0c82bc5-c7ba-4225-b473-827155cd5358","Type":"ContainerStarted","Data":"6431593da6e9c85eca6a0d617b4c598908c19e094152f6444c7880a01043bb01"}
Nov 02 13:44:13.821390 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:13.821331    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-cfc8c476c-6jvqg" podStartSLOduration=1.821313854 podStartE2EDuration="1.821313854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:13.820178149 +0000 UTC m=+44.571482587" watchObservedRunningTime="2026-11-02 13:44:13.821313854 +0000 UTC m=+44.572618348"
Nov 02 13:44:14.756754 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:14.756707    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt" podUID="3fa80f89-f24d-4f86-940c-864dbf177a5c" containerName="registry" containerID="cri-o://3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5" gracePeriod=30
Nov 02 13:44:15.053694 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.053668    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:44:15.130465 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.130436    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted\") pod \"3fa80f89-f24d-4f86-940c-864dbf177a5c\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") "
Nov 02 13:44:15.130607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.130504    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token\") pod \"3fa80f89-f24d-4f86-940c-864dbf177a5c\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") "
Nov 02 13:44:15.130607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.130537    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration\") pod \"3fa80f89-f24d-4f86-940c-864dbf177a5c\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") "
Nov 02 13:44:15.130607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.130557    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") pod \"3fa80f89-f24d-4f86-940c-864dbf177a5c\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") "
Nov 02 13:44:15.130607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.130584    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4\") pod \"3fa80f89-f24d-4f86-940c-864dbf177a5c\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") "
Nov 02 13:44:15.130852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.130627    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca\") pod \"3fa80f89-f24d-4f86-940c-864dbf177a5c\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") "
Nov 02 13:44:15.130852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.130670    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates\") pod \"3fa80f89-f24d-4f86-940c-864dbf177a5c\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") "
Nov 02 13:44:15.131213 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.131160    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca" pod "3fa80f89-f24d-4f86-940c-864dbf177a5c" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:15.131475 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.131415    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates" pod "3fa80f89-f24d-4f86-940c-864dbf177a5c" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:15.131475 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.131436    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets\") pod \"3fa80f89-f24d-4f86-940c-864dbf177a5c\" (UID: \"3fa80f89-f24d-4f86-940c-864dbf177a5c\") "
Nov 02 13:44:15.131759 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.131728    5034 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-trusted-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:15.131928 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.131913    5034 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-certificates\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:15.133555 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.133527    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4" pod "3fa80f89-f24d-4f86-940c-864dbf177a5c" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c"). InnerVolumeSpecName "kube-api-access-q4fv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:15.133694 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.133653    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls" pod "3fa80f89-f24d-4f86-940c-864dbf177a5c" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:15.133751 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.133708    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration" pod "3fa80f89-f24d-4f86-940c-864dbf177a5c" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:15.133979 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.133942    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token" pod "3fa80f89-f24d-4f86-940c-864dbf177a5c" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:15.134085 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.133983    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets" pod "3fa80f89-f24d-4f86-940c-864dbf177a5c" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:15.137870 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.137848    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted" pod "3fa80f89-f24d-4f86-940c-864dbf177a5c" (UID: "3fa80f89-f24d-4f86-940c-864dbf177a5c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:15.233183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.233155    5034 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-image-registry-private-configuration\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:15.233183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.233179    5034 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-registry-tls\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:15.233332 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.233194    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4fv4\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-kube-api-access-q4fv4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:15.233332 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.233209    5034 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3fa80f89-f24d-4f86-940c-864dbf177a5c-installation-pull-secrets\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:15.233332 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.233223    5034 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3fa80f89-f24d-4f86-940c-864dbf177a5c-ca-trust-extracted\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:15.233332 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.233236    5034 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3fa80f89-f24d-4f86-940c-864dbf177a5c-bound-sa-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:15.765494 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.765463    5034 generic.go:356] "Generic (PLEG): container finished" podID="3fa80f89-f24d-4f86-940c-864dbf177a5c" containerID="3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5" exitCode=0
Nov 02 13:44:15.765878 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.765519    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt"
Nov 02 13:44:15.765878 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.765550    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt" event={"ID":"3fa80f89-f24d-4f86-940c-864dbf177a5c","Type":"ContainerDied","Data":"3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5"}
Nov 02 13:44:15.765878 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.765628    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d555fdbf5-mnstt" event={"ID":"3fa80f89-f24d-4f86-940c-864dbf177a5c","Type":"ContainerDied","Data":"901a0f1565ded0308d83487d185d54a54fbb53c714cd3c8ba3df90d291cf0b7d"}
Nov 02 13:44:15.765878 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.765653    5034 scope.go:122] "RemoveContainer" containerID="3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5"
Nov 02 13:44:15.774189 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.774170    5034 scope.go:122] "RemoveContainer" containerID="3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5"
Nov 02 13:44:15.774462 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:44:15.774440    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5\": container with ID starting with 3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5 not found: ID does not exist" containerID="3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5"
Nov 02 13:44:15.774525 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.774470    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5"} err="failed to get container status \"3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5\": rpc error: code = NotFound desc = could not find container \"3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5\": container with ID starting with 3f56e4b5c4576b7190904a4fb4f13099050963bfb15e7fb01501eea1f3358fa5 not found: ID does not exist"
Nov 02 13:44:15.832761 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.832735    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5d555fdbf5-mnstt"]
Nov 02 13:44:15.850180 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:15.850152    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5d555fdbf5-mnstt"]
Nov 02 13:44:17.392174 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:17.392141    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa80f89-f24d-4f86-940c-864dbf177a5c" path="/var/lib/kubelet/pods/3fa80f89-f24d-4f86-940c-864dbf177a5c/volumes"
Nov 02 13:44:18.412799 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:44:18.412748    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:44:21.970298 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:21.970271    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q59bv-5ptkf"]
Nov 02 13:44:21.970724 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:21.970441    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-q59bv-5ptkf" podUID="99fc1fe5-048b-4196-b66c-771710cd73c0" containerName="insights-gathering" containerID="cri-o://c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b" gracePeriod=30
Nov 02 13:44:22.298310 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.298280    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:44:22.386215 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.386151    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path\") pod \"99fc1fe5-048b-4196-b66c-771710cd73c0\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") "
Nov 02 13:44:22.386215 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.386195    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql\" (UniqueName: \"kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql\") pod \"99fc1fe5-048b-4196-b66c-771710cd73c0\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") "
Nov 02 13:44:22.386411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.386272    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle\") pod \"99fc1fe5-048b-4196-b66c-771710cd73c0\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") "
Nov 02 13:44:22.386411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.386292    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle\") pod \"99fc1fe5-048b-4196-b66c-771710cd73c0\" (UID: \"99fc1fe5-048b-4196-b66c-771710cd73c0\") "
Nov 02 13:44:22.386586 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.386549    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path" pod "99fc1fe5-048b-4196-b66c-771710cd73c0" (UID: "99fc1fe5-048b-4196-b66c-771710cd73c0"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:22.386711 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.386604    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle" pod "99fc1fe5-048b-4196-b66c-771710cd73c0" (UID: "99fc1fe5-048b-4196-b66c-771710cd73c0"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:22.386711 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.386667    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle" pod "99fc1fe5-048b-4196-b66c-771710cd73c0" (UID: "99fc1fe5-048b-4196-b66c-771710cd73c0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:22.388653 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.388625    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql" pod "99fc1fe5-048b-4196-b66c-771710cd73c0" (UID: "99fc1fe5-048b-4196-b66c-771710cd73c0"). InnerVolumeSpecName "kube-api-access-thqql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:22.487807 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.487765    5034 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/99fc1fe5-048b-4196-b66c-771710cd73c0-archives-path\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:22.487807 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.487807    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thqql\" (UniqueName: \"kubernetes.io/projected/99fc1fe5-048b-4196-b66c-771710cd73c0-kube-api-access-thqql\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:22.487942 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.487819    5034 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-service-ca-bundle\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:22.487942 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.487827    5034 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99fc1fe5-048b-4196-b66c-771710cd73c0-trusted-ca-bundle\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:22.628396 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.628366    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:22.628531 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.628412    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:22.633291 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.633268    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:22.797394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.797363    5034 generic.go:356] "Generic (PLEG): container finished" podID="99fc1fe5-048b-4196-b66c-771710cd73c0" containerID="c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b" exitCode=2
Nov 02 13:44:22.797532 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.797410    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q59bv-5ptkf"
Nov 02 13:44:22.797532 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.797453    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q59bv-5ptkf" event={"ID":"99fc1fe5-048b-4196-b66c-771710cd73c0","Type":"ContainerDied","Data":"c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b"}
Nov 02 13:44:22.797532 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.797497    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q59bv-5ptkf" event={"ID":"99fc1fe5-048b-4196-b66c-771710cd73c0","Type":"ContainerDied","Data":"6cd201012c26fed6351658b222ea0788f7f9813fcc30d7408e37e40c4b6fa864"}
Nov 02 13:44:22.797532 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.797520    5034 scope.go:122] "RemoveContainer" containerID="c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b"
Nov 02 13:44:22.801762 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.801744    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:44:22.806833 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.806814    5034 scope.go:122] "RemoveContainer" containerID="c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b"
Nov 02 13:44:22.807079 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:44:22.807062    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b\": container with ID starting with c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b not found: ID does not exist" containerID="c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b"
Nov 02 13:44:22.807137 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.807087    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b"} err="failed to get container status \"c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b\": rpc error: code = NotFound desc = could not find container \"c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b\": container with ID starting with c397f0aee1769139f102f17740426b806fe013eda7c3d8a5638766dec53afc5b not found: ID does not exist"
Nov 02 13:44:22.888351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.888319    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858dfd774b-4m89s"]
Nov 02 13:44:22.901571 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.901544    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-q59bv-5ptkf"]
Nov 02 13:44:22.910286 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:22.910259    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-q59bv-5ptkf"]
Nov 02 13:44:23.392363 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:23.392325    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99fc1fe5-048b-4196-b66c-771710cd73c0" path="/var/lib/kubelet/pods/99fc1fe5-048b-4196-b66c-771710cd73c0/volumes"
Nov 02 13:44:37.221307 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.221244    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5456f4ccf6-gsh5j" podUID="f9cbabf2-f039-4466-97dc-601c191e8d9b" containerName="console" containerID="cri-o://d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383" gracePeriod=15
Nov 02 13:44:37.504383 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.504354    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:37.589947 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.589920    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-trusted-ca-bundle\") pod \"f9cbabf2-f039-4466-97dc-601c191e8d9b\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") "
Nov 02 13:44:37.590084 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.589952    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-serving-cert\") pod \"f9cbabf2-f039-4466-97dc-601c191e8d9b\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") "
Nov 02 13:44:37.590084 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.589980    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-service-ca\") pod \"f9cbabf2-f039-4466-97dc-601c191e8d9b\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") "
Nov 02 13:44:37.590084 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590055    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-config\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-config\") pod \"f9cbabf2-f039-4466-97dc-601c191e8d9b\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") "
Nov 02 13:44:37.590301 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590098    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9cbabf2-f039-4466-97dc-601c191e8d9b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9cbabf2-f039-4466-97dc-601c191e8d9b-tmp\") pod \"f9cbabf2-f039-4466-97dc-601c191e8d9b\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") "
Nov 02 13:44:37.590301 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590125    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9cbabf2-f039-4466-97dc-601c191e8d9b-kube-api-access-p4w62\" (UniqueName: \"kubernetes.io/projected/f9cbabf2-f039-4466-97dc-601c191e8d9b-kube-api-access-p4w62\") pod \"f9cbabf2-f039-4466-97dc-601c191e8d9b\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") "
Nov 02 13:44:37.590301 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590154    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-oauth-config\") pod \"f9cbabf2-f039-4466-97dc-601c191e8d9b\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") "
Nov 02 13:44:37.590301 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590245    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-oauth-serving-cert\") pod \"f9cbabf2-f039-4466-97dc-601c191e8d9b\" (UID: \"f9cbabf2-f039-4466-97dc-601c191e8d9b\") "
Nov 02 13:44:37.590509 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590386    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9cbabf2-f039-4466-97dc-601c191e8d9b-tmp" pod "f9cbabf2-f039-4466-97dc-601c191e8d9b" (UID: "f9cbabf2-f039-4466-97dc-601c191e8d9b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:37.590509 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590419    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-trusted-ca-bundle" pod "f9cbabf2-f039-4466-97dc-601c191e8d9b" (UID: "f9cbabf2-f039-4466-97dc-601c191e8d9b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:37.590615 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590529    5034 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f9cbabf2-f039-4466-97dc-601c191e8d9b-tmp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:37.590615 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590541    5034 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-trusted-ca-bundle\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:37.590615 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590538    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-config" pod "f9cbabf2-f039-4466-97dc-601c191e8d9b" (UID: "f9cbabf2-f039-4466-97dc-601c191e8d9b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:37.590615 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590571    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-service-ca" pod "f9cbabf2-f039-4466-97dc-601c191e8d9b" (UID: "f9cbabf2-f039-4466-97dc-601c191e8d9b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:37.590892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.590869    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-oauth-serving-cert" pod "f9cbabf2-f039-4466-97dc-601c191e8d9b" (UID: "f9cbabf2-f039-4466-97dc-601c191e8d9b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:37.592284 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.592258    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-serving-cert" pod "f9cbabf2-f039-4466-97dc-601c191e8d9b" (UID: "f9cbabf2-f039-4466-97dc-601c191e8d9b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:37.592384 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.592319    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9cbabf2-f039-4466-97dc-601c191e8d9b-kube-api-access-p4w62" pod "f9cbabf2-f039-4466-97dc-601c191e8d9b" (UID: "f9cbabf2-f039-4466-97dc-601c191e8d9b"). InnerVolumeSpecName "kube-api-access-p4w62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:37.592424 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.592392    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-oauth-config" pod "f9cbabf2-f039-4466-97dc-601c191e8d9b" (UID: "f9cbabf2-f039-4466-97dc-601c191e8d9b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:37.691031 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.691007    5034 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-oauth-serving-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:37.691031 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.691029    5034 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-serving-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:37.691165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.691039    5034 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:37.691165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.691048    5034 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-config\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:37.691165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.691056    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4w62\" (UniqueName: \"kubernetes.io/projected/f9cbabf2-f039-4466-97dc-601c191e8d9b-kube-api-access-p4w62\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:37.691165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.691065    5034 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9cbabf2-f039-4466-97dc-601c191e8d9b-console-oauth-config\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:37.870592 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.870568    5034 generic.go:356] "Generic (PLEG): container finished" podID="f9cbabf2-f039-4466-97dc-601c191e8d9b" containerID="d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383" exitCode=0
Nov 02 13:44:37.870713 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.870630    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5456f4ccf6-gsh5j"
Nov 02 13:44:37.870713 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.870642    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5456f4ccf6-gsh5j" event={"ID":"f9cbabf2-f039-4466-97dc-601c191e8d9b","Type":"ContainerDied","Data":"d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383"}
Nov 02 13:44:37.870713 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.870704    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5456f4ccf6-gsh5j" event={"ID":"f9cbabf2-f039-4466-97dc-601c191e8d9b","Type":"ContainerDied","Data":"9e4a952c6329267437b5cd0d52f9dc8adbe118f20874d5fcb2c1ca0aed6193f0"}
Nov 02 13:44:37.870841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.870724    5034 scope.go:122] "RemoveContainer" containerID="d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383"
Nov 02 13:44:37.879122 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.879105    5034 scope.go:122] "RemoveContainer" containerID="d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383"
Nov 02 13:44:37.879387 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:44:37.879360    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383\": container with ID starting with d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383 not found: ID does not exist" containerID="d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383"
Nov 02 13:44:37.879479 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.879386    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383"} err="failed to get container status \"d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383\": rpc error: code = NotFound desc = could not find container \"d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383\": container with ID starting with d242c59496ebfc892ec7e4ec26269b28b3b011dc0c87c86177ab935d08aad383 not found: ID does not exist"
Nov 02 13:44:37.923670 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.923642    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5456f4ccf6-gsh5j"]
Nov 02 13:44:37.936507 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:37.936489    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5456f4ccf6-gsh5j"]
Nov 02 13:44:38.745859 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:38.745808    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk" podUID="633c4858-9dc5-4049-aec3-8a14171652ae" containerName="registry" containerID="cri-o://bd87bffa132f9b5504404f362f74d971cc572ed005070825494acce7ac91cdd4" gracePeriod=30
Nov 02 13:44:38.877358 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:38.877329    5034 generic.go:356] "Generic (PLEG): container finished" podID="8646ac2d-cfd2-4b7c-929a-906815e6551f" containerID="0da50c343fbf19b365fec4ea13a72fd4739a077381e9dde3db9e465610655134" exitCode=0
Nov 02 13:44:38.877461 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:38.877394    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-f79wg" event={"ID":"8646ac2d-cfd2-4b7c-929a-906815e6551f","Type":"ContainerDied","Data":"0da50c343fbf19b365fec4ea13a72fd4739a077381e9dde3db9e465610655134"}
Nov 02 13:44:38.877461 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:38.877452    5034 scope.go:122] "RemoveContainer" containerID="aafbeb05fba56682112c11251207e14667fb860ac195b82aa9d7d61bf56a15a6"
Nov 02 13:44:38.877774 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:38.877755    5034 scope.go:122] "RemoveContainer" containerID="0da50c343fbf19b365fec4ea13a72fd4739a077381e9dde3db9e465610655134"
Nov 02 13:44:38.878058 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:44:38.878035    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-f79wg_openshift-insights(8646ac2d-cfd2-4b7c-929a-906815e6551f)\"" pod="openshift-insights/insights-operator-7c4986d449-f79wg" podUID="8646ac2d-cfd2-4b7c-929a-906815e6551f"
Nov 02 13:44:38.879066 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:38.879047    5034 generic.go:356] "Generic (PLEG): container finished" podID="633c4858-9dc5-4049-aec3-8a14171652ae" containerID="bd87bffa132f9b5504404f362f74d971cc572ed005070825494acce7ac91cdd4" exitCode=0
Nov 02 13:44:38.879184 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:38.879121    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk" event={"ID":"633c4858-9dc5-4049-aec3-8a14171652ae","Type":"ContainerDied","Data":"bd87bffa132f9b5504404f362f74d971cc572ed005070825494acce7ac91cdd4"}
Nov 02 13:44:39.044258 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.044231    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:44:39.102212 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102175    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") pod \"633c4858-9dc5-4049-aec3-8a14171652ae\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") "
Nov 02 13:44:39.102357 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102223    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq\") pod \"633c4858-9dc5-4049-aec3-8a14171652ae\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") "
Nov 02 13:44:39.102357 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102271    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration\") pod \"633c4858-9dc5-4049-aec3-8a14171652ae\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") "
Nov 02 13:44:39.102357 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102291    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca\") pod \"633c4858-9dc5-4049-aec3-8a14171652ae\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") "
Nov 02 13:44:39.102357 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102316    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets\") pod \"633c4858-9dc5-4049-aec3-8a14171652ae\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") "
Nov 02 13:44:39.102575 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102434    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token\") pod \"633c4858-9dc5-4049-aec3-8a14171652ae\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") "
Nov 02 13:44:39.102575 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102471    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates\") pod \"633c4858-9dc5-4049-aec3-8a14171652ae\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") "
Nov 02 13:44:39.102575 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102556    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted\") pod \"633c4858-9dc5-4049-aec3-8a14171652ae\" (UID: \"633c4858-9dc5-4049-aec3-8a14171652ae\") "
Nov 02 13:44:39.102996 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102755    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca" pod "633c4858-9dc5-4049-aec3-8a14171652ae" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:39.102996 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.102881    5034 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-trusted-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:39.103314 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.103203    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates" pod "633c4858-9dc5-4049-aec3-8a14171652ae" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:39.104817 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.104739    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq" pod "633c4858-9dc5-4049-aec3-8a14171652ae" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae"). InnerVolumeSpecName "kube-api-access-dbrcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:39.105122 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.105100    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls" pod "633c4858-9dc5-4049-aec3-8a14171652ae" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:39.105242 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.105178    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration" pod "633c4858-9dc5-4049-aec3-8a14171652ae" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:39.105313 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.105276    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets" pod "633c4858-9dc5-4049-aec3-8a14171652ae" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:39.105481 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.105460    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token" pod "633c4858-9dc5-4049-aec3-8a14171652ae" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:39.109949 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.109925    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted" pod "633c4858-9dc5-4049-aec3-8a14171652ae" (UID: "633c4858-9dc5-4049-aec3-8a14171652ae"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:39.204018 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.203986    5034 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-image-registry-private-configuration\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:39.204018 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.204010    5034 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/633c4858-9dc5-4049-aec3-8a14171652ae-installation-pull-secrets\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:39.204018 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.204021    5034 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-bound-sa-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:39.204236 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.204029    5034 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/633c4858-9dc5-4049-aec3-8a14171652ae-registry-certificates\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:39.204236 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.204038    5034 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/633c4858-9dc5-4049-aec3-8a14171652ae-ca-trust-extracted\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:39.204236 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.204046    5034 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-registry-tls\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:39.204236 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.204054    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrcq\" (UniqueName: \"kubernetes.io/projected/633c4858-9dc5-4049-aec3-8a14171652ae-kube-api-access-dbrcq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:39.392156 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.392101    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9cbabf2-f039-4466-97dc-601c191e8d9b" path="/var/lib/kubelet/pods/f9cbabf2-f039-4466-97dc-601c191e8d9b/volumes"
Nov 02 13:44:39.884851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.884826    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk"
Nov 02 13:44:39.885269 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.884845    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8c6787bcd-psbdk" event={"ID":"633c4858-9dc5-4049-aec3-8a14171652ae","Type":"ContainerDied","Data":"0b02c25bf9c1d0684da7a1cee4f2d42fd7607b285fb8570e3332f8ea6d4e5ac9"}
Nov 02 13:44:39.885269 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.884903    5034 scope.go:122] "RemoveContainer" containerID="bd87bffa132f9b5504404f362f74d971cc572ed005070825494acce7ac91cdd4"
Nov 02 13:44:39.951358 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.951328    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8c6787bcd-psbdk"]
Nov 02 13:44:39.964384 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:39.964364    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8c6787bcd-psbdk"]
Nov 02 13:44:41.392077 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:41.392045    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633c4858-9dc5-4049-aec3-8a14171652ae" path="/var/lib/kubelet/pods/633c4858-9dc5-4049-aec3-8a14171652ae/volumes"
Nov 02 13:44:47.910153 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:47.910115    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-858dfd774b-4m89s" podUID="133f2f2b-96a4-4bc3-a844-9f19a068dbf7" containerName="console" containerID="cri-o://9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021" gracePeriod=15
Nov 02 13:44:48.187578 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.187515    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:44:48.267483 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.267450    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-service-ca\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-service-ca\") pod \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") "
Nov 02 13:44:48.267483 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.267486    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-kube-api-access-9mtjt\" (UniqueName: \"kubernetes.io/projected/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-kube-api-access-9mtjt\") pod \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") "
Nov 02 13:44:48.267661 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.267520    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-oauth-config\") pod \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") "
Nov 02 13:44:48.267661 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.267591    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-config\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-config\") pod \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") "
Nov 02 13:44:48.267661 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.267632    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-serving-cert\") pod \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") "
Nov 02 13:44:48.267841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.267678    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-tmp\") pod \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") "
Nov 02 13:44:48.267841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.267734    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-oauth-serving-cert\") pod \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\" (UID: \"133f2f2b-96a4-4bc3-a844-9f19a068dbf7\") "
Nov 02 13:44:48.267958 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.267911    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-service-ca" pod "133f2f2b-96a4-4bc3-a844-9f19a068dbf7" (UID: "133f2f2b-96a4-4bc3-a844-9f19a068dbf7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:48.268055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.268033    5034 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.268276 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.268250    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-config" pod "133f2f2b-96a4-4bc3-a844-9f19a068dbf7" (UID: "133f2f2b-96a4-4bc3-a844-9f19a068dbf7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:48.268276 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.268259    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-oauth-serving-cert" pod "133f2f2b-96a4-4bc3-a844-9f19a068dbf7" (UID: "133f2f2b-96a4-4bc3-a844-9f19a068dbf7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:48.268572 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.268543    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-tmp" pod "133f2f2b-96a4-4bc3-a844-9f19a068dbf7" (UID: "133f2f2b-96a4-4bc3-a844-9f19a068dbf7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:48.269825 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.269769    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-serving-cert" pod "133f2f2b-96a4-4bc3-a844-9f19a068dbf7" (UID: "133f2f2b-96a4-4bc3-a844-9f19a068dbf7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:48.269916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.269836    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-kube-api-access-9mtjt" pod "133f2f2b-96a4-4bc3-a844-9f19a068dbf7" (UID: "133f2f2b-96a4-4bc3-a844-9f19a068dbf7"). InnerVolumeSpecName "kube-api-access-9mtjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:48.269916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.269839    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-oauth-config" pod "133f2f2b-96a4-4bc3-a844-9f19a068dbf7" (UID: "133f2f2b-96a4-4bc3-a844-9f19a068dbf7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:44:48.368548 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.368525    5034 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-oauth-serving-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.368548 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.368547    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtjt\" (UniqueName: \"kubernetes.io/projected/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-kube-api-access-9mtjt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.368682 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.368562    5034 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-oauth-config\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.368682 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.368575    5034 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-config\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.368682 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.368589    5034 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-console-serving-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.368682 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.368601    5034 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/133f2f2b-96a4-4bc3-a844-9f19a068dbf7-tmp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:48.411179 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:44:48.411151    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:44:48.923970 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.923936    5034 generic.go:356] "Generic (PLEG): container finished" podID="133f2f2b-96a4-4bc3-a844-9f19a068dbf7" containerID="9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021" exitCode=0
Nov 02 13:44:48.924447 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.923981    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858dfd774b-4m89s"
Nov 02 13:44:48.924447 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.924016    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858dfd774b-4m89s" event={"ID":"133f2f2b-96a4-4bc3-a844-9f19a068dbf7","Type":"ContainerDied","Data":"9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021"}
Nov 02 13:44:48.924447 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.924065    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858dfd774b-4m89s" event={"ID":"133f2f2b-96a4-4bc3-a844-9f19a068dbf7","Type":"ContainerDied","Data":"0b91b3146d34bc6efd63e62223aaaf27d1c46c467d375c75cf3d454751612a1f"}
Nov 02 13:44:48.924447 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.924086    5034 scope.go:122] "RemoveContainer" containerID="9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021"
Nov 02 13:44:48.933939 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.933922    5034 scope.go:122] "RemoveContainer" containerID="9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021"
Nov 02 13:44:48.934217 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:44:48.934198    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021\": container with ID starting with 9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021 not found: ID does not exist" containerID="9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021"
Nov 02 13:44:48.934285 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.934223    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021"} err="failed to get container status \"9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021\": rpc error: code = NotFound desc = could not find container \"9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021\": container with ID starting with 9da3d845f99f7dd36afb91b48bb868a4151a966fb2a4f94996f933973bc1e021 not found: ID does not exist"
Nov 02 13:44:48.977445 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.977411    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858dfd774b-4m89s"]
Nov 02 13:44:48.990063 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:48.990032    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-858dfd774b-4m89s"]
Nov 02 13:44:49.391554 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:49.391524    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133f2f2b-96a4-4bc3-a844-9f19a068dbf7" path="/var/lib/kubelet/pods/133f2f2b-96a4-4bc3-a844-9f19a068dbf7/volumes"
Nov 02 13:44:50.387720 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:50.387687    5034 scope.go:122] "RemoveContainer" containerID="0da50c343fbf19b365fec4ea13a72fd4739a077381e9dde3db9e465610655134"
Nov 02 13:44:50.935423 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:50.935389    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-f79wg" event={"ID":"8646ac2d-cfd2-4b7c-929a-906815e6551f","Type":"ContainerStarted","Data":"d5375d4ba611116abc0f354d6f220d26c8b81b6da0634085ac6d33d1bfabf8dd"}
Nov 02 13:44:51.002935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.002903    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-66rdf-vjshb"]
Nov 02 13:44:51.003149 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.003121    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-66rdf-vjshb" podUID="cfa38906-422d-4391-bfc0-a696866c7a20" containerName="insights-gathering" containerID="cri-o://22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa" gracePeriod=30
Nov 02 13:44:51.294063 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.294037    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:44:51.395719 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.395684    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-trusted-ca-bundle\") pod \"cfa38906-422d-4391-bfc0-a696866c7a20\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") "
Nov 02 13:44:51.396181 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.395842    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfa38906-422d-4391-bfc0-a696866c7a20-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cfa38906-422d-4391-bfc0-a696866c7a20-archives-path\") pod \"cfa38906-422d-4391-bfc0-a696866c7a20\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") "
Nov 02 13:44:51.396181 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.395887    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-service-ca-bundle\") pod \"cfa38906-422d-4391-bfc0-a696866c7a20\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") "
Nov 02 13:44:51.396181 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.395939    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa38906-422d-4391-bfc0-a696866c7a20-kube-api-access-dm95z\" (UniqueName: \"kubernetes.io/projected/cfa38906-422d-4391-bfc0-a696866c7a20-kube-api-access-dm95z\") pod \"cfa38906-422d-4391-bfc0-a696866c7a20\" (UID: \"cfa38906-422d-4391-bfc0-a696866c7a20\") "
Nov 02 13:44:51.396181 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.395989    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-trusted-ca-bundle" pod "cfa38906-422d-4391-bfc0-a696866c7a20" (UID: "cfa38906-422d-4391-bfc0-a696866c7a20"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:51.396376 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.396203    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfa38906-422d-4391-bfc0-a696866c7a20-archives-path" pod "cfa38906-422d-4391-bfc0-a696866c7a20" (UID: "cfa38906-422d-4391-bfc0-a696866c7a20"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:44:51.396376 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.396278    5034 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-trusted-ca-bundle\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:51.396376 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.396295    5034 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cfa38906-422d-4391-bfc0-a696866c7a20-archives-path\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:51.396376 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.396347    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-service-ca-bundle" pod "cfa38906-422d-4391-bfc0-a696866c7a20" (UID: "cfa38906-422d-4391-bfc0-a696866c7a20"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:44:51.400181 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.400155    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa38906-422d-4391-bfc0-a696866c7a20-kube-api-access-dm95z" pod "cfa38906-422d-4391-bfc0-a696866c7a20" (UID: "cfa38906-422d-4391-bfc0-a696866c7a20"). InnerVolumeSpecName "kube-api-access-dm95z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:44:51.497334 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.497306    5034 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cfa38906-422d-4391-bfc0-a696866c7a20-service-ca-bundle\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:51.497334 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.497329    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm95z\" (UniqueName: \"kubernetes.io/projected/cfa38906-422d-4391-bfc0-a696866c7a20-kube-api-access-dm95z\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:44:51.940815 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.940772    5034 generic.go:356] "Generic (PLEG): container finished" podID="cfa38906-422d-4391-bfc0-a696866c7a20" containerID="22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa" exitCode=2
Nov 02 13:44:51.940970 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.940848    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-66rdf-vjshb"
Nov 02 13:44:51.940970 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.940864    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-66rdf-vjshb" event={"ID":"cfa38906-422d-4391-bfc0-a696866c7a20","Type":"ContainerDied","Data":"22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa"}
Nov 02 13:44:51.940970 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.940915    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-66rdf-vjshb" event={"ID":"cfa38906-422d-4391-bfc0-a696866c7a20","Type":"ContainerDied","Data":"35756f88dc70196277f3aa7d5c82f6500d63bdbc31015bbff876b20e87e186a5"}
Nov 02 13:44:51.940970 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.940935    5034 scope.go:122] "RemoveContainer" containerID="22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa"
Nov 02 13:44:51.949413 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.949398    5034 scope.go:122] "RemoveContainer" containerID="22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa"
Nov 02 13:44:51.949681 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:44:51.949659    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa\": container with ID starting with 22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa not found: ID does not exist" containerID="22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa"
Nov 02 13:44:51.949736 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.949688    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa"} err="failed to get container status \"22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa\": rpc error: code = NotFound desc = could not find container \"22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa\": container with ID starting with 22727c792fda14b3a7d6158ffc0690c03f23fa7d4987af91582aa5bb2e2787aa not found: ID does not exist"
Nov 02 13:44:51.993697 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.993675    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-66rdf-vjshb"]
Nov 02 13:44:51.999809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:51.999790    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-66rdf-vjshb"]
Nov 02 13:44:53.391344 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:53.391313    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa38906-422d-4391-bfc0-a696866c7a20" path="/var/lib/kubelet/pods/cfa38906-422d-4391-bfc0-a696866c7a20/volumes"
Nov 02 13:44:56.903817 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.903771    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fd56bb96d-5xgbl"]
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904074    5034 cpu_manager.go:414] "containerMap: removing container" podUID="133f2f2b-96a4-4bc3-a844-9f19a068dbf7" containerName="console"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904084    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133f2f2b-96a4-4bc3-a844-9f19a068dbf7" containerName="console"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904100    5034 cpu_manager.go:414] "containerMap: removing container" podUID="99fc1fe5-048b-4196-b66c-771710cd73c0" containerName="insights-gathering"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904105    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99fc1fe5-048b-4196-b66c-771710cd73c0" containerName="insights-gathering"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904114    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cfa38906-422d-4391-bfc0-a696866c7a20" containerName="insights-gathering"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904119    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa38906-422d-4391-bfc0-a696866c7a20" containerName="insights-gathering"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904127    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f9cbabf2-f039-4466-97dc-601c191e8d9b" containerName="console"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904132    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9cbabf2-f039-4466-97dc-601c191e8d9b" containerName="console"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904147    5034 cpu_manager.go:414] "containerMap: removing container" podUID="633c4858-9dc5-4049-aec3-8a14171652ae" containerName="registry"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904151    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633c4858-9dc5-4049-aec3-8a14171652ae" containerName="registry"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904159    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3fa80f89-f24d-4f86-940c-864dbf177a5c" containerName="registry"
Nov 02 13:44:56.904183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904163    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa80f89-f24d-4f86-940c-864dbf177a5c" containerName="registry"
Nov 02 13:44:56.904536 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904216    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="99fc1fe5-048b-4196-b66c-771710cd73c0" containerName="insights-gathering"
Nov 02 13:44:56.904536 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904224    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa38906-422d-4391-bfc0-a696866c7a20" containerName="insights-gathering"
Nov 02 13:44:56.904536 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904231    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9cbabf2-f039-4466-97dc-601c191e8d9b" containerName="console"
Nov 02 13:44:56.904536 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904239    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="633c4858-9dc5-4049-aec3-8a14171652ae" containerName="registry"
Nov 02 13:44:56.904536 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904245    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa80f89-f24d-4f86-940c-864dbf177a5c" containerName="registry"
Nov 02 13:44:56.904536 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.904251    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="133f2f2b-96a4-4bc3-a844-9f19a068dbf7" containerName="console"
Nov 02 13:44:56.906908 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.906892    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:56.936189 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:56.936162    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fd56bb96d-5xgbl"]
Nov 02 13:44:57.037380 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.037354    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-oauth-serving-cert\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.037540 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.037391    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-config\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.037540 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.037408    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ngz7\" (UniqueName: \"kubernetes.io/projected/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-kube-api-access-7ngz7\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.037540 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.037470    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-serving-cert\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.037660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.037540    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-tmp\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.037660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.037574    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-oauth-config\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.037660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.037590    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-service-ca\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.037660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.037609    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-trusted-ca-bundle\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.138087 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.138060    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-tmp\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.138217 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.138098    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-oauth-config\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.138217 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.138114    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-service-ca\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.138217 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.138131    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-trusted-ca-bundle\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.138217 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.138171    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-oauth-serving-cert\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.138217 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.138205    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-config\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.138446 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.138227    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ngz7\" (UniqueName: \"kubernetes.io/projected/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-kube-api-access-7ngz7\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.138446 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.138264    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-serving-cert\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.138635 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.138603    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-tmp\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.139073 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.139054    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-service-ca\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.139163 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.139079    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-config\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.139210 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.139157    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-trusted-ca-bundle\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.139210 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.139181    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-oauth-serving-cert\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.140756 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.140733    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-serving-cert\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.140860 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.140813    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-oauth-config\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.162652 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.162605    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ngz7\" (UniqueName: \"kubernetes.io/projected/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-kube-api-access-7ngz7\") pod \"console-7fd56bb96d-5xgbl\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") " pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.218676 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.218651    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:44:57.519354 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.519331    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fd56bb96d-5xgbl"]
Nov 02 13:44:57.520980 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:44:57.520944    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c16ac8c_aa5c_4450_8a9d_367cac7e2c92.slice/crio-e8bd967c2394e16efdb5110d207d2f6e491642839e56942ae9162215ff8ed6ec WatchSource:0}: Error finding container e8bd967c2394e16efdb5110d207d2f6e491642839e56942ae9162215ff8ed6ec: Status 404 returned error can't find the container with id e8bd967c2394e16efdb5110d207d2f6e491642839e56942ae9162215ff8ed6ec
Nov 02 13:44:57.968363 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.968274    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd56bb96d-5xgbl" event={"ID":"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92","Type":"ContainerStarted","Data":"7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4"}
Nov 02 13:44:57.968363 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:57.968353    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd56bb96d-5xgbl" event={"ID":"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92","Type":"ContainerStarted","Data":"e8bd967c2394e16efdb5110d207d2f6e491642839e56942ae9162215ff8ed6ec"}
Nov 02 13:44:58.014416 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:44:58.014368    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7fd56bb96d-5xgbl" podStartSLOduration=2.014355925 podStartE2EDuration="2.014355925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:44:58.013743431 +0000 UTC m=+88.765047869" watchObservedRunningTime="2026-11-02 13:44:58.014355925 +0000 UTC m=+88.765660364"
Nov 02 13:45:07.219269 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:07.219209    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:45:07.219269 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:07.219285    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:45:07.224163 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:07.224134    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:45:08.016952 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:08.016921    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:45:08.092908 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:08.092875    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cfc8c476c-6jvqg"]
Nov 02 13:45:18.413359 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:45:18.413319    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:45:33.113331 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.113258    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-cfc8c476c-6jvqg" podUID="a0c82bc5-c7ba-4225-b473-827155cd5358" containerName="console" containerID="cri-o://6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3" gracePeriod=15
Nov 02 13:45:33.367025 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.366953    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:45:33.429790 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.429743    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-oauth-serving-cert\") pod \"a0c82bc5-c7ba-4225-b473-827155cd5358\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") "
Nov 02 13:45:33.429971 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.429801    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-oauth-config\") pod \"a0c82bc5-c7ba-4225-b473-827155cd5358\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") "
Nov 02 13:45:33.429971 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.429897    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-service-ca\") pod \"a0c82bc5-c7ba-4225-b473-827155cd5358\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") "
Nov 02 13:45:33.429971 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.429955    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0c82bc5-c7ba-4225-b473-827155cd5358-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0c82bc5-c7ba-4225-b473-827155cd5358-tmp\") pod \"a0c82bc5-c7ba-4225-b473-827155cd5358\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") "
Nov 02 13:45:33.430144 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.429989    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c82bc5-c7ba-4225-b473-827155cd5358-kube-api-access-f2t2r\" (UniqueName: \"kubernetes.io/projected/a0c82bc5-c7ba-4225-b473-827155cd5358-kube-api-access-f2t2r\") pod \"a0c82bc5-c7ba-4225-b473-827155cd5358\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") "
Nov 02 13:45:33.430144 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430016    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-trusted-ca-bundle\") pod \"a0c82bc5-c7ba-4225-b473-827155cd5358\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") "
Nov 02 13:45:33.430144 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430070    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-console-config\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-console-config\") pod \"a0c82bc5-c7ba-4225-b473-827155cd5358\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") "
Nov 02 13:45:33.430144 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430112    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-serving-cert\") pod \"a0c82bc5-c7ba-4225-b473-827155cd5358\" (UID: \"a0c82bc5-c7ba-4225-b473-827155cd5358\") "
Nov 02 13:45:33.430343 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430175    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-oauth-serving-cert" pod "a0c82bc5-c7ba-4225-b473-827155cd5358" (UID: "a0c82bc5-c7ba-4225-b473-827155cd5358"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:33.430343 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430285    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0c82bc5-c7ba-4225-b473-827155cd5358-tmp" pod "a0c82bc5-c7ba-4225-b473-827155cd5358" (UID: "a0c82bc5-c7ba-4225-b473-827155cd5358"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:45:33.430343 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430293    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-service-ca" pod "a0c82bc5-c7ba-4225-b473-827155cd5358" (UID: "a0c82bc5-c7ba-4225-b473-827155cd5358"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:33.430705 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430453    5034 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:45:33.430705 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430475    5034 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0c82bc5-c7ba-4225-b473-827155cd5358-tmp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:45:33.430705 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430489    5034 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-oauth-serving-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:45:33.430705 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430559    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-console-config" pod "a0c82bc5-c7ba-4225-b473-827155cd5358" (UID: "a0c82bc5-c7ba-4225-b473-827155cd5358"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:33.430992 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.430760    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-trusted-ca-bundle" pod "a0c82bc5-c7ba-4225-b473-827155cd5358" (UID: "a0c82bc5-c7ba-4225-b473-827155cd5358"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:45:33.432224 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.432181    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-oauth-config" pod "a0c82bc5-c7ba-4225-b473-827155cd5358" (UID: "a0c82bc5-c7ba-4225-b473-827155cd5358"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:45:33.432482 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.432454    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-serving-cert" pod "a0c82bc5-c7ba-4225-b473-827155cd5358" (UID: "a0c82bc5-c7ba-4225-b473-827155cd5358"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:45:33.432605 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.432584    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c82bc5-c7ba-4225-b473-827155cd5358-kube-api-access-f2t2r" pod "a0c82bc5-c7ba-4225-b473-827155cd5358" (UID: "a0c82bc5-c7ba-4225-b473-827155cd5358"). InnerVolumeSpecName "kube-api-access-f2t2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:45:33.530892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.530851    5034 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-oauth-config\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:45:33.530892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.530883    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2t2r\" (UniqueName: \"kubernetes.io/projected/a0c82bc5-c7ba-4225-b473-827155cd5358-kube-api-access-f2t2r\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:45:33.530892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.530895    5034 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-trusted-ca-bundle\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:45:33.530892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.530904    5034 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a0c82bc5-c7ba-4225-b473-827155cd5358-console-config\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:45:33.531164 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:33.530912    5034 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a0c82bc5-c7ba-4225-b473-827155cd5358-console-serving-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:45:34.122129 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:34.122094    5034 generic.go:356] "Generic (PLEG): container finished" podID="a0c82bc5-c7ba-4225-b473-827155cd5358" containerID="6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3" exitCode=0
Nov 02 13:45:34.122573 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:34.122148    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cfc8c476c-6jvqg"
Nov 02 13:45:34.122573 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:34.122160    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfc8c476c-6jvqg" event={"ID":"a0c82bc5-c7ba-4225-b473-827155cd5358","Type":"ContainerDied","Data":"6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3"}
Nov 02 13:45:34.122573 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:34.122212    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cfc8c476c-6jvqg" event={"ID":"a0c82bc5-c7ba-4225-b473-827155cd5358","Type":"ContainerDied","Data":"6431593da6e9c85eca6a0d617b4c598908c19e094152f6444c7880a01043bb01"}
Nov 02 13:45:34.122573 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:34.122239    5034 scope.go:122] "RemoveContainer" containerID="6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3"
Nov 02 13:45:34.131602 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:34.131582    5034 scope.go:122] "RemoveContainer" containerID="6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3"
Nov 02 13:45:34.131940 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:45:34.131918    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3\": container with ID starting with 6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3 not found: ID does not exist" containerID="6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3"
Nov 02 13:45:34.131988 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:34.131951    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3"} err="failed to get container status \"6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3\": rpc error: code = NotFound desc = could not find container \"6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3\": container with ID starting with 6fbe31f96cf683a0fe2c75d5867df33e4c01ee2a197674a5212eaff53b4c9eb3 not found: ID does not exist"
Nov 02 13:45:34.175679 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:34.175644    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cfc8c476c-6jvqg"]
Nov 02 13:45:34.188448 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:34.188413    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cfc8c476c-6jvqg"]
Nov 02 13:45:35.391878 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:35.391836    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c82bc5-c7ba-4225-b473-827155cd5358" path="/var/lib/kubelet/pods/a0c82bc5-c7ba-4225-b473-827155cd5358/volumes"
Nov 02 13:45:48.411056 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:45:48.410973    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:45:50.896410 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.896376    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b7f467c7c-lmsjl"]
Nov 02 13:45:50.896848 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.896694    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a0c82bc5-c7ba-4225-b473-827155cd5358" containerName="console"
Nov 02 13:45:50.896848 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.896704    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c82bc5-c7ba-4225-b473-827155cd5358" containerName="console"
Nov 02 13:45:50.896848 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.896790    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c82bc5-c7ba-4225-b473-827155cd5358" containerName="console"
Nov 02 13:45:50.898618 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.898601    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:50.927977 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.927952    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7f467c7c-lmsjl"]
Nov 02 13:45:50.962239 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.962207    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-trusted-ca-bundle\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:50.962239 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.962244    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-oauth-serving-cert\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:50.962464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.962269    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85b0ee54-727d-4550-90b3-3df145901123-console-serving-cert\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:50.962464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.962315    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-console-config\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:50.962464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.962409    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85b0ee54-727d-4550-90b3-3df145901123-tmp\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:50.962599 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.962478    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h2zf\" (UniqueName: \"kubernetes.io/projected/85b0ee54-727d-4550-90b3-3df145901123-kube-api-access-8h2zf\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:50.962599 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.962506    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-service-ca\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:50.962599 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:50.962526    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85b0ee54-727d-4550-90b3-3df145901123-console-oauth-config\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.063492 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.063463    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-trusted-ca-bundle\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.063633 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.063503    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-oauth-serving-cert\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.063633 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.063534    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85b0ee54-727d-4550-90b3-3df145901123-console-serving-cert\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.063633 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.063565    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-console-config\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.063633 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.063589    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85b0ee54-727d-4550-90b3-3df145901123-tmp\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.063633 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.063612    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h2zf\" (UniqueName: \"kubernetes.io/projected/85b0ee54-727d-4550-90b3-3df145901123-kube-api-access-8h2zf\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.063915 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.063638    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-service-ca\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.063915 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.063668    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85b0ee54-727d-4550-90b3-3df145901123-console-oauth-config\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.064125 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.064094    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85b0ee54-727d-4550-90b3-3df145901123-tmp\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.064396 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.064375    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-oauth-serving-cert\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.064396 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.064385    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-console-config\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.064510 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.064418    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-service-ca\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.064574 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.064557    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b0ee54-727d-4550-90b3-3df145901123-trusted-ca-bundle\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.066576 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.066554    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/85b0ee54-727d-4550-90b3-3df145901123-console-oauth-config\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.066817 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.066766    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/85b0ee54-727d-4550-90b3-3df145901123-console-serving-cert\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.087856 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.087835    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h2zf\" (UniqueName: \"kubernetes.io/projected/85b0ee54-727d-4550-90b3-3df145901123-kube-api-access-8h2zf\") pod \"console-7b7f467c7c-lmsjl\" (UID: \"85b0ee54-727d-4550-90b3-3df145901123\") " pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.208409 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.208338    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:45:51.509667 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:51.509642    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7f467c7c-lmsjl"]
Nov 02 13:45:51.510496 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:45:51.510471    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b0ee54_727d_4550_90b3_3df145901123.slice/crio-013f844579255da21a6a3d64d48b9aa126c12b10c59cf6aa6cfc08363131cb3e WatchSource:0}: Error finding container 013f844579255da21a6a3d64d48b9aa126c12b10c59cf6aa6cfc08363131cb3e: Status 404 returned error can't find the container with id 013f844579255da21a6a3d64d48b9aa126c12b10c59cf6aa6cfc08363131cb3e
Nov 02 13:45:52.195467 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:52.195431    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7f467c7c-lmsjl" event={"ID":"85b0ee54-727d-4550-90b3-3df145901123","Type":"ContainerStarted","Data":"d3144cbb69dbad2330093ae3dbad19dec6601a0703be22ef32ed73956cf7efd2"}
Nov 02 13:45:52.195871 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:52.195484    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7f467c7c-lmsjl" event={"ID":"85b0ee54-727d-4550-90b3-3df145901123","Type":"ContainerStarted","Data":"013f844579255da21a6a3d64d48b9aa126c12b10c59cf6aa6cfc08363131cb3e"}
Nov 02 13:45:52.240722 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:45:52.240671    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b7f467c7c-lmsjl" podStartSLOduration=2.240656194 podStartE2EDuration="2.240656194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:45:52.240101174 +0000 UTC m=+142.991405622" watchObservedRunningTime="2026-11-02 13:45:52.240656194 +0000 UTC m=+142.991960631"
Nov 02 13:46:01.208908 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:01.208864    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:46:01.209389 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:01.208966    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:46:01.213950 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:01.213929    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:46:01.234024 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:01.233999    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b7f467c7c-lmsjl"
Nov 02 13:46:01.336773 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:01.336739    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fd56bb96d-5xgbl"]
Nov 02 13:46:18.411375 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:46:18.411340    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:46:26.357905 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.357866    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7fd56bb96d-5xgbl" podUID="7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" containerName="console" containerID="cri-o://7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4" gracePeriod=15
Nov 02 13:46:26.597241 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.597217    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:46:26.734512 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.734488    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-service-ca\") pod \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") "
Nov 02 13:46:26.734661 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.734521    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-config\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-config\") pod \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") "
Nov 02 13:46:26.734661 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.734558    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-trusted-ca-bundle\") pod \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") "
Nov 02 13:46:26.734661 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.734582    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-kube-api-access-7ngz7\" (UniqueName: \"kubernetes.io/projected/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-kube-api-access-7ngz7\") pod \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") "
Nov 02 13:46:26.734865 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.734693    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-oauth-config\") pod \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") "
Nov 02 13:46:26.734865 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.734741    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-oauth-serving-cert\") pod \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") "
Nov 02 13:46:26.734979 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.734868    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-tmp\") pod \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") "
Nov 02 13:46:26.734979 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.734902    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-serving-cert\") pod \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\" (UID: \"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92\") "
Nov 02 13:46:26.735083 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.735054    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-service-ca" pod "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" (UID: "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:46:26.735083 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.735061    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-config" pod "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" (UID: "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:46:26.735188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.735118    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-trusted-ca-bundle" pod "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" (UID: "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:46:26.735188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.735142    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-oauth-serving-cert" pod "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" (UID: "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:46:26.735290 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.735202    5034 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-trusted-ca-bundle\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:46:26.735339 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.735320    5034 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:46:26.735389 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.735337    5034 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-config\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:46:26.735389 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.735331    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-tmp" pod "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" (UID: "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:46:26.736970 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.736952    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-serving-cert" pod "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" (UID: "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:46:26.737242 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.737219    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-kube-api-access-7ngz7" pod "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" (UID: "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92"). InnerVolumeSpecName "kube-api-access-7ngz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:46:26.737242 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.737229    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-oauth-config" pod "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" (UID: "7c16ac8c-aa5c-4450-8a9d-367cac7e2c92"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:46:26.836128 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.836104    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ngz7\" (UniqueName: \"kubernetes.io/projected/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-kube-api-access-7ngz7\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:46:26.836128 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.836126    5034 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-oauth-config\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:46:26.836258 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.836135    5034 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-oauth-serving-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:46:26.836258 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.836143    5034 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-tmp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:46:26.836258 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:26.836153    5034 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92-console-serving-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:46:27.332433 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:27.332398    5034 generic.go:356] "Generic (PLEG): container finished" podID="7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" containerID="7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4" exitCode=0
Nov 02 13:46:27.332595 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:27.332449    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fd56bb96d-5xgbl"
Nov 02 13:46:27.332595 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:27.332471    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd56bb96d-5xgbl" event={"ID":"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92","Type":"ContainerDied","Data":"7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4"}
Nov 02 13:46:27.332595 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:27.332515    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fd56bb96d-5xgbl" event={"ID":"7c16ac8c-aa5c-4450-8a9d-367cac7e2c92","Type":"ContainerDied","Data":"e8bd967c2394e16efdb5110d207d2f6e491642839e56942ae9162215ff8ed6ec"}
Nov 02 13:46:27.332595 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:27.332536    5034 scope.go:122] "RemoveContainer" containerID="7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4"
Nov 02 13:46:27.341227 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:27.341210    5034 scope.go:122] "RemoveContainer" containerID="7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4"
Nov 02 13:46:27.341514 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:46:27.341495    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4\": container with ID starting with 7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4 not found: ID does not exist" containerID="7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4"
Nov 02 13:46:27.341564 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:27.341525    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4"} err="failed to get container status \"7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4\": rpc error: code = NotFound desc = could not find container \"7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4\": container with ID starting with 7bc987ae3751b5afa8846ad1ce39f6040b1b38dca4c295cd2bb780d50b14ace4 not found: ID does not exist"
Nov 02 13:46:27.391351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:27.391331    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fd56bb96d-5xgbl"]
Nov 02 13:46:27.399082 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:27.399064    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fd56bb96d-5xgbl"]
Nov 02 13:46:29.393008 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:46:29.392977    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" path="/var/lib/kubelet/pods/7c16ac8c-aa5c-4450-8a9d-367cac7e2c92/volumes"
Nov 02 13:46:48.412447 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:46:48.412411    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:47:18.411762 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:47:18.411681    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:47:48.411631 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:47:48.411595    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:47:59.361934 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.361902    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"]
Nov 02 13:47:59.362420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.362403    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" containerName="console"
Nov 02 13:47:59.362475 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.362422    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" containerName="console"
Nov 02 13:47:59.362528 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.362510    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c16ac8c-aa5c-4450-8a9d-367cac7e2c92" containerName="console"
Nov 02 13:47:59.365838 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.365816    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.383732 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.383708    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-27gvt\""
Nov 02 13:47:59.396377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.396351    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 13:47:59.398428 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.398409    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"]
Nov 02 13:47:59.514936 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.514900    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.515066 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.514942    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.515375 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.515345    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.515481 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.515417    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1af079-7c3c-444f-a426-b716bde35270-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.515481 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.515446    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4txx\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-kube-api-access-c4txx\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.515568 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.515482    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.515568 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.515533    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.515669 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.515564    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1af079-7c3c-444f-a426-b716bde35270-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.515669 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.515620    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.528041 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.528011    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"]
Nov 02 13:47:59.531321 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.531300    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.562633 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.562613    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-mfmh5\""
Nov 02 13:47:59.565224 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.565201    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"]
Nov 02 13:47:59.616700 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.616641    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.616700 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.616671    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.616700 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.616694    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.616918 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.616713    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.616918 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.616740    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.616918 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.616890    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.617027 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.616957    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.617027 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.616980    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1af079-7c3c-444f-a426-b716bde35270-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.617027 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.616997    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4txx\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-kube-api-access-c4txx\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.617027 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617016    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.617199 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617037    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.617199 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617079    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.617199 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617122    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.617199 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617145    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.617199 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617182    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.617392 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617237    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.617392 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617315    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7rgv\" (UniqueName: \"kubernetes.io/projected/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-kube-api-access-g7rgv\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.617503 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617485    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.617546 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617527    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1af079-7c3c-444f-a426-b716bde35270-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.617582 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617552    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.617637 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617584    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.617637 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.617581    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.618194 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.618162    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1af079-7c3c-444f-a426-b716bde35270-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.619257 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.619235    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.619409 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.619391    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1af079-7c3c-444f-a426-b716bde35270-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.646736 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.646707    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.646875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.646861    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4txx\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-kube-api-access-c4txx\") pod \"maas-default-gateway-openshift-default-6886b96f9-9f9xp\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.678464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.678440    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:47:59.718708 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.718675    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.718708 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.718715    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.719087 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.718740    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.719087 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.718789    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.719087 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.718820    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.719087 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.718853    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7rgv\" (UniqueName: \"kubernetes.io/projected/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-kube-api-access-g7rgv\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.719288 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.719200    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.719288 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.719221    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.719575 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.719550    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.719955 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.719936    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.720032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.720003    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.720094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.720041    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.720336 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.720316    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.720392 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.720367    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.721130 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.721109    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.721674 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.721655    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.747799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.747746    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.748209 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.748188    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7rgv\" (UniqueName: \"kubernetes.io/projected/50a1f6b2-1feb-4aaa-bbb7-f224ce55f893-kube-api-access-g7rgv\") pod \"openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg\" (UID: \"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.844511 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.844478    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:47:59.995131 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:47:59.995100    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"]
Nov 02 13:47:59.995588 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:47:59.995565    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1af079_7c3c_444f_a426_b716bde35270.slice/crio-d67e91b95c1190fc36392c089514bc39896e460ea21a14a5a86f57403f4621bc WatchSource:0}: Error finding container d67e91b95c1190fc36392c089514bc39896e460ea21a14a5a86f57403f4621bc: Status 404 returned error can't find the container with id d67e91b95c1190fc36392c089514bc39896e460ea21a14a5a86f57403f4621bc
Nov 02 13:48:00.141083 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:00.141026    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"]
Nov 02 13:48:00.141428 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:48:00.141401    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a1f6b2_1feb_4aaa_bbb7_f224ce55f893.slice/crio-48569a0716e77c5df0de53b147d3ebe5fd8be40df9c4798270e502b6108813bd WatchSource:0}: Error finding container 48569a0716e77c5df0de53b147d3ebe5fd8be40df9c4798270e502b6108813bd: Status 404 returned error can't find the container with id 48569a0716e77c5df0de53b147d3ebe5fd8be40df9c4798270e502b6108813bd
Nov 02 13:48:00.685503 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:00.685466    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg" event={"ID":"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893","Type":"ContainerStarted","Data":"48569a0716e77c5df0de53b147d3ebe5fd8be40df9c4798270e502b6108813bd"}
Nov 02 13:48:00.686597 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:00.686576    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" event={"ID":"cb1af079-7c3c-444f-a426-b716bde35270","Type":"ContainerStarted","Data":"d67e91b95c1190fc36392c089514bc39896e460ea21a14a5a86f57403f4621bc"}
Nov 02 13:48:03.088055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.088021    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:48:03.088340 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.088089    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:48:03.088340 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.088119    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:48:03.202660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.202619    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:48:03.202770 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.202707    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:48:03.202770 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.202756    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:48:03.701584 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.701544    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg" event={"ID":"50a1f6b2-1feb-4aaa-bbb7-f224ce55f893","Type":"ContainerStarted","Data":"45093df99e5dbb477cca1d18d8ad7e4bcb6256ca249994f5675cb6f680443d83"}
Nov 02 13:48:03.703011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.702986    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" event={"ID":"cb1af079-7c3c-444f-a426-b716bde35270","Type":"ContainerStarted","Data":"daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e"}
Nov 02 13:48:03.773400 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.773351    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg" podStartSLOduration=1.714829022 podStartE2EDuration="4.773338239s" totalImagesPullingTime="3.058509217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:03.760105346 +0000 UTC m=+274.511409784" watchObservedRunningTime="2026-11-02 13:48:03.773338239 +0000 UTC m=+274.524642667"
Nov 02 13:48:03.805916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.805875    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podStartSLOduration=1.716052514 podStartE2EDuration="4.805861669s" totalImagesPullingTime="3.089809155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:47:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:03.805418977 +0000 UTC m=+274.556723416" watchObservedRunningTime="2026-11-02 13:48:03.805861669 +0000 UTC m=+274.557166108"
Nov 02 13:48:03.845516 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:03.845494    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:48:04.679370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:04.679340    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:48:04.683728 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:04.683703    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:48:04.706980 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:04.706957    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:48:04.707589 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:04.707571    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:48:04.850024 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:04.849993    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:48:05.710346 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:05.710316    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:48:05.711091 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:05.711071    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg"
Nov 02 13:48:18.411984 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:48:18.411944    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:48:29.409868 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:29.409837    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 13:48:29.410684 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:29.410667    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 13:48:40.503794 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.503692    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"]
Nov 02 13:48:40.513839 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.513818    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.533478 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.533455    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"]
Nov 02 13:48:40.544466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.544441    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 13:48:40.544466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.544445    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 13:48:40.559988 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.559964    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 13:48:40.560147 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.560129    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 13:48:40.560234 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.560177    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-drxxd\""
Nov 02 13:48:40.561040 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.561028    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 13:48:40.626543 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.626517    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11379774-bf53-4fe3-97de-04c77a4a6f7b-tmp\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.626650 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.626592    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/11379774-bf53-4fe3-97de-04c77a4a6f7b-metrics-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.626650 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.626625    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11379774-bf53-4fe3-97de-04c77a4a6f7b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.626966 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.626657    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11379774-bf53-4fe3-97de-04c77a4a6f7b-manifests\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.626966 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.626731    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fpcm\" (UniqueName: \"kubernetes.io/projected/11379774-bf53-4fe3-97de-04c77a4a6f7b-kube-api-access-7fpcm\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.626966 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.626765    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11379774-bf53-4fe3-97de-04c77a4a6f7b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.727821 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.727773    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11379774-bf53-4fe3-97de-04c77a4a6f7b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.727947 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.727845    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11379774-bf53-4fe3-97de-04c77a4a6f7b-manifests\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.727947 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.727873    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fpcm\" (UniqueName: \"kubernetes.io/projected/11379774-bf53-4fe3-97de-04c77a4a6f7b-kube-api-access-7fpcm\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.727947 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.727905    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11379774-bf53-4fe3-97de-04c77a4a6f7b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.728095 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.727962    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11379774-bf53-4fe3-97de-04c77a4a6f7b-tmp\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.728095 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.728047    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/11379774-bf53-4fe3-97de-04c77a4a6f7b-metrics-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.728372 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.728343    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/11379774-bf53-4fe3-97de-04c77a4a6f7b-manifests\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.730400 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.730372    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/11379774-bf53-4fe3-97de-04c77a4a6f7b-tmp\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.730637 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.730610    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/11379774-bf53-4fe3-97de-04c77a4a6f7b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.730728 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.730683    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/11379774-bf53-4fe3-97de-04c77a4a6f7b-webhook-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.731758 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.731735    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/11379774-bf53-4fe3-97de-04c77a4a6f7b-metrics-cert\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.764634 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.764580    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fpcm\" (UniqueName: \"kubernetes.io/projected/11379774-bf53-4fe3-97de-04c77a4a6f7b-kube-api-access-7fpcm\") pod \"opendatahub-operator-controller-manager-f4bb4dd59-cvjlv\" (UID: \"11379774-bf53-4fe3-97de-04c77a4a6f7b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:40.830914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:40.830880    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:41.183192 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:41.183165    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"]
Nov 02 13:48:41.183485 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:48:41.183463    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11379774_bf53_4fe3_97de_04c77a4a6f7b.slice/crio-b4de6e555c806f712c45fae4ea7dcf57513d1aaffbdecfd8e1491944f290fd85 WatchSource:0}: Error finding container b4de6e555c806f712c45fae4ea7dcf57513d1aaffbdecfd8e1491944f290fd85: Status 404 returned error can't find the container with id b4de6e555c806f712c45fae4ea7dcf57513d1aaffbdecfd8e1491944f290fd85
Nov 02 13:48:41.185118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:41.185100    5034 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:48:41.873522 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:41.873481    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv" event={"ID":"11379774-bf53-4fe3-97de-04c77a4a6f7b","Type":"ContainerStarted","Data":"b4de6e555c806f712c45fae4ea7dcf57513d1aaffbdecfd8e1491944f290fd85"}
Nov 02 13:48:44.888603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:44.888567    5034 generic.go:356] "Generic (PLEG): container finished" podID="11379774-bf53-4fe3-97de-04c77a4a6f7b" containerID="fa00b270e0d4f6d87a9532999f1b7a06056b09d8178ec231c0181e03ea1ce206" exitCode=0
Nov 02 13:48:44.889107 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:44.888688    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv" event={"ID":"11379774-bf53-4fe3-97de-04c77a4a6f7b","Type":"ContainerDied","Data":"fa00b270e0d4f6d87a9532999f1b7a06056b09d8178ec231c0181e03ea1ce206"}
Nov 02 13:48:45.894603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:45.894561    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv" event={"ID":"11379774-bf53-4fe3-97de-04c77a4a6f7b","Type":"ContainerStarted","Data":"907151baafc76828815c7b69f17a74b09715f7f7161bb541960c8be645b41e3e"}
Nov 02 13:48:45.895139 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:45.894683    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:48:46.059196 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:46.059144    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv" podStartSLOduration=2.533852613 podStartE2EDuration="6.0591253s" totalImagesPullingTime="3.098014944s" totalInitContainerRuntime="427.257743ms" isStatefulPod=false podCreationTimestamp="2026-11-02 13:48:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:48:46.045476115 +0000 UTC m=+316.796780547" watchObservedRunningTime="2026-11-02 13:48:46.0591253 +0000 UTC m=+316.810429740"
Nov 02 13:48:48.412011 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:48:48.411977    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:48:56.901953 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:48:56.901919    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-f4bb4dd59-cvjlv"
Nov 02 13:49:01.694963 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:01.694921    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"]
Nov 02 13:49:01.701311 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:01.701289    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"
Nov 02 13:49:01.730565 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:01.730543    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-4xpv5\""
Nov 02 13:49:01.735812 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:01.735729    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 13:49:01.743389 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:01.743370    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 13:49:01.745348 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:01.745324    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"]
Nov 02 13:49:01.797561 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:01.797530    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4d7\" (UniqueName: \"kubernetes.io/projected/fc17acb2-e694-4585-8c2b-3fed98a4df14-kube-api-access-jz4d7\") pod \"authorino-operator-7b7bb4bc77-cscl8\" (UID: \"fc17acb2-e694-4585-8c2b-3fed98a4df14\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"
Nov 02 13:49:01.898405 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:01.898373    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4d7\" (UniqueName: \"kubernetes.io/projected/fc17acb2-e694-4585-8c2b-3fed98a4df14-kube-api-access-jz4d7\") pod \"authorino-operator-7b7bb4bc77-cscl8\" (UID: \"fc17acb2-e694-4585-8c2b-3fed98a4df14\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"
Nov 02 13:49:01.949530 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:01.947037    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4d7\" (UniqueName: \"kubernetes.io/projected/fc17acb2-e694-4585-8c2b-3fed98a4df14-kube-api-access-jz4d7\") pod \"authorino-operator-7b7bb4bc77-cscl8\" (UID: \"fc17acb2-e694-4585-8c2b-3fed98a4df14\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"
Nov 02 13:49:02.014555 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:02.014527    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"
Nov 02 13:49:02.355993 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:02.355960    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"]
Nov 02 13:49:02.356245 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:49:02.356221    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc17acb2_e694_4585_8c2b_3fed98a4df14.slice/crio-a48d72e6e97b834307cf6242b067cd51a07e32d0088201f65d67626514080701 WatchSource:0}: Error finding container a48d72e6e97b834307cf6242b067cd51a07e32d0088201f65d67626514080701: Status 404 returned error can't find the container with id a48d72e6e97b834307cf6242b067cd51a07e32d0088201f65d67626514080701
Nov 02 13:49:02.966227 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:02.966193    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8" event={"ID":"fc17acb2-e694-4585-8c2b-3fed98a4df14","Type":"ContainerStarted","Data":"a48d72e6e97b834307cf6242b067cd51a07e32d0088201f65d67626514080701"}
Nov 02 13:49:04.977140 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:04.977061    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8" event={"ID":"fc17acb2-e694-4585-8c2b-3fed98a4df14","Type":"ContainerStarted","Data":"92ca603029e0c4c7cd1fab5e09c3d4c9aa948b36bdb3e6b5efc8895494f8a198"}
Nov 02 13:49:04.977514 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:04.977202    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"
Nov 02 13:49:05.026709 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:05.026656    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8" podStartSLOduration=1.724918878 podStartE2EDuration="4.026641997s" totalImagesPullingTime="2.301723119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:05.02496814 +0000 UTC m=+335.776272578" watchObservedRunningTime="2026-11-02 13:49:05.026641997 +0000 UTC m=+335.777946435"
Nov 02 13:49:06.212537 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.212503    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"]
Nov 02 13:49:06.216097 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.216081    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:06.233317 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.233296    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"]
Nov 02 13:49:06.253534 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.253515    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m76wd\""
Nov 02 13:49:06.337995 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.337971    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2d6c\" (UniqueName: \"kubernetes.io/projected/58393844-0472-4a85-ba31-9586f62ea2c2-kube-api-access-k2d6c\") pod \"kuadrant-operator-controller-manager-678d87766f-bwz2h\" (UID: \"58393844-0472-4a85-ba31-9586f62ea2c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:06.338119 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.338027    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58393844-0472-4a85-ba31-9586f62ea2c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bwz2h\" (UID: \"58393844-0472-4a85-ba31-9586f62ea2c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:06.439069 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.439044    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2d6c\" (UniqueName: \"kubernetes.io/projected/58393844-0472-4a85-ba31-9586f62ea2c2-kube-api-access-k2d6c\") pod \"kuadrant-operator-controller-manager-678d87766f-bwz2h\" (UID: \"58393844-0472-4a85-ba31-9586f62ea2c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:06.439204 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.439105    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58393844-0472-4a85-ba31-9586f62ea2c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bwz2h\" (UID: \"58393844-0472-4a85-ba31-9586f62ea2c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:06.439433 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.439414    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58393844-0472-4a85-ba31-9586f62ea2c2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bwz2h\" (UID: \"58393844-0472-4a85-ba31-9586f62ea2c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:06.464671 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.464622    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2d6c\" (UniqueName: \"kubernetes.io/projected/58393844-0472-4a85-ba31-9586f62ea2c2-kube-api-access-k2d6c\") pod \"kuadrant-operator-controller-manager-678d87766f-bwz2h\" (UID: \"58393844-0472-4a85-ba31-9586f62ea2c2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:06.529119 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.529091    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:06.865663 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.865627    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"]
Nov 02 13:49:06.878304 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:49:06.878275    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58393844_0472_4a85_ba31_9586f62ea2c2.slice/crio-820e6b58586736a5194c327191978a9d685e183c1993d499cf54f907735372c3 WatchSource:0}: Error finding container 820e6b58586736a5194c327191978a9d685e183c1993d499cf54f907735372c3: Status 404 returned error can't find the container with id 820e6b58586736a5194c327191978a9d685e183c1993d499cf54f907735372c3
Nov 02 13:49:06.986790 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:06.986740    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h" event={"ID":"58393844-0472-4a85-ba31-9586f62ea2c2","Type":"ContainerStarted","Data":"820e6b58586736a5194c327191978a9d685e183c1993d499cf54f907735372c3"}
Nov 02 13:49:11.008500 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:11.008462    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h" event={"ID":"58393844-0472-4a85-ba31-9586f62ea2c2","Type":"ContainerStarted","Data":"4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f"}
Nov 02 13:49:11.008913 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:11.008670    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:11.088919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:11.088862    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h" podStartSLOduration=1.500767496 podStartE2EDuration="5.088847968s" totalImagesPullingTime="3.588080472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:11.075499197 +0000 UTC m=+341.826803647" watchObservedRunningTime="2026-11-02 13:49:11.088847968 +0000 UTC m=+341.840152403"
Nov 02 13:49:15.984117 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:15.984085    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-cscl8"
Nov 02 13:49:18.413449 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:49:18.413412    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:49:22.015443 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:22.015413    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 13:49:26.943804 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:26.943744    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"]
Nov 02 13:49:26.951006 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:26.950986    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:26.979570 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:26.979549    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-ztk5s\""
Nov 02 13:49:27.008879 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.008849    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znw7g\" (UniqueName: \"kubernetes.io/projected/4983a055-8bd9-43ad-ba61-20cdb90ed101-kube-api-access-znw7g\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.009030 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.008916    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.009030 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.009011    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.009149 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.009121    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.009203 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.009178    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4983a055-8bd9-43ad-ba61-20cdb90ed101-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.009241 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.009219    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.009294 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.009256    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.009294 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.009283    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.009375 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.009363    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.014535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.014512    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"]
Nov 02 13:49:27.110013 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.109980    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110026    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4983a055-8bd9-43ad-ba61-20cdb90ed101-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110071    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110092    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110107    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110153    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110481 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110209    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znw7g\" (UniqueName: \"kubernetes.io/projected/4983a055-8bd9-43ad-ba61-20cdb90ed101-kube-api-access-znw7g\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110481 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110258    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110481 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110293    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110704 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110527    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110988 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110917    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.110988 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.110919    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4983a055-8bd9-43ad-ba61-20cdb90ed101-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.111126 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.111083    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.111178 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.111145    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.112399 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.112378    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.113345 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.113320    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.136181 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.136152    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4983a055-8bd9-43ad-ba61-20cdb90ed101-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.136355 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.136338    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znw7g\" (UniqueName: \"kubernetes.io/projected/4983a055-8bd9-43ad-ba61-20cdb90ed101-kube-api-access-znw7g\") pod \"data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv\" (UID: \"4983a055-8bd9-43ad-ba61-20cdb90ed101\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.263979 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.263946    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:27.606500 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.606471    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"]
Nov 02 13:49:27.606882 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:49:27.606860    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4983a055_8bd9_43ad_ba61_20cdb90ed101.slice/crio-5b83f03e9d188b9b74f54ff200cf04241990e2316eea71b81a04e9c4b0f30617 WatchSource:0}: Error finding container 5b83f03e9d188b9b74f54ff200cf04241990e2316eea71b81a04e9c4b0f30617: Status 404 returned error can't find the container with id 5b83f03e9d188b9b74f54ff200cf04241990e2316eea71b81a04e9c4b0f30617
Nov 02 13:49:27.609102 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.609067    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:49:27.609174 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.609135    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:49:27.609210 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:27.609175    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:49:28.087059 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.087019    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv" event={"ID":"4983a055-8bd9-43ad-ba61-20cdb90ed101","Type":"ContainerStarted","Data":"e95891ca1f72d37367c58c9a66db48775856b76e86539b8aba0c6c25b7561364"}
Nov 02 13:49:28.087554 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.087091    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv" event={"ID":"4983a055-8bd9-43ad-ba61-20cdb90ed101","Type":"ContainerStarted","Data":"5b83f03e9d188b9b74f54ff200cf04241990e2316eea71b81a04e9c4b0f30617"}
Nov 02 13:49:28.144461 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.144412    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv" podStartSLOduration=2.144398412 podStartE2EDuration="2.144398412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:28.143932595 +0000 UTC m=+358.895237045" watchObservedRunningTime="2026-11-02 13:49:28.144398412 +0000 UTC m=+358.895702924"
Nov 02 13:49:28.264535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.264493    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:28.269377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.269352    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:28.886134 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.886098    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwqlh"]
Nov 02 13:49:28.889813 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.889790    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-qwqlh"
Nov 02 13:49:28.908065 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.908025    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c5j78\""
Nov 02 13:49:28.909652 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.909628    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwqlh"]
Nov 02 13:49:28.927118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:28.927089    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4l99\" (UniqueName: \"kubernetes.io/projected/d4518e06-3103-4ae1-b027-54faab03b1a3-kube-api-access-f4l99\") pod \"authorino-57f7f8659f-qwqlh\" (UID: \"d4518e06-3103-4ae1-b027-54faab03b1a3\") " pod="kuadrant-system/authorino-57f7f8659f-qwqlh"
Nov 02 13:49:29.028570 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.028539    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4l99\" (UniqueName: \"kubernetes.io/projected/d4518e06-3103-4ae1-b027-54faab03b1a3-kube-api-access-f4l99\") pod \"authorino-57f7f8659f-qwqlh\" (UID: \"d4518e06-3103-4ae1-b027-54faab03b1a3\") " pod="kuadrant-system/authorino-57f7f8659f-qwqlh"
Nov 02 13:49:29.054483 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.054453    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4l99\" (UniqueName: \"kubernetes.io/projected/d4518e06-3103-4ae1-b027-54faab03b1a3-kube-api-access-f4l99\") pod \"authorino-57f7f8659f-qwqlh\" (UID: \"d4518e06-3103-4ae1-b027-54faab03b1a3\") " pod="kuadrant-system/authorino-57f7f8659f-qwqlh"
Nov 02 13:49:29.091567 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.091542    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:29.092591 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.092567    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv"
Nov 02 13:49:29.210144 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.210078    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-qwqlh"
Nov 02 13:49:29.223072 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.223047    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"]
Nov 02 13:49:29.226681 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.226659    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.250706 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.250674    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"]
Nov 02 13:49:29.260561 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.260539    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 13:49:29.260793 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.260539    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-p26zs\""
Nov 02 13:49:29.260873 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.260656    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 13:49:29.279868 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.279850    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:49:29.280003 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.279850    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 13:49:29.280261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.280241    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 13:49:29.330295 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.330268    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-metrics-certs\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.330433 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.330315    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-manager-config\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.330433 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.330333    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p4s\" (UniqueName: \"kubernetes.io/projected/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-kube-api-access-w4p4s\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.330523 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.330460    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-cert\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.431218 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.431187    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-metrics-certs\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.431369 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.431252    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-manager-config\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.431369 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.431304    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p4s\" (UniqueName: \"kubernetes.io/projected/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-kube-api-access-w4p4s\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.431449 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.431402    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-cert\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.445754 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.445733    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 13:49:29.445911 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.445801    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 13:49:29.445911 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.445800    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 13:49:29.452702 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.452679    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-manager-config\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.454368 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.454349    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-metrics-certs\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.454429 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.454396    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-cert\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.467564 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.467514    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 13:49:29.487356 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.487333    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 13:49:29.495831 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.495809    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p4s\" (UniqueName: \"kubernetes.io/projected/8b0d7ea3-7cc8-4308-a764-28f01e34c5c2-kube-api-access-w4p4s\") pod \"jobset-controller-manager-869dc4b47b-pbmzx\" (UID: \"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2\") " pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.519133 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.519112    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwqlh"]
Nov 02 13:49:29.519859 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:49:29.519835    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4518e06_3103_4ae1_b027_54faab03b1a3.slice/crio-b18e0f6926b23ce4465698a571c0c3063244bcf67ad0bb82839b18435b6c5ff5 WatchSource:0}: Error finding container b18e0f6926b23ce4465698a571c0c3063244bcf67ad0bb82839b18435b6c5ff5: Status 404 returned error can't find the container with id b18e0f6926b23ce4465698a571c0c3063244bcf67ad0bb82839b18435b6c5ff5
Nov 02 13:49:29.558725 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.558696    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-p26zs\""
Nov 02 13:49:29.565303 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.565280    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:29.873144 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:29.873119    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"]
Nov 02 13:49:29.873587 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:49:29.873566    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0d7ea3_7cc8_4308_a764_28f01e34c5c2.slice/crio-a7aa9583e80e1d30e6c104223063dfb5ec6b47d94d83d2c53905aa254ef84718 WatchSource:0}: Error finding container a7aa9583e80e1d30e6c104223063dfb5ec6b47d94d83d2c53905aa254ef84718: Status 404 returned error can't find the container with id a7aa9583e80e1d30e6c104223063dfb5ec6b47d94d83d2c53905aa254ef84718
Nov 02 13:49:30.096440 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:30.096402    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx" event={"ID":"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2","Type":"ContainerStarted","Data":"a7aa9583e80e1d30e6c104223063dfb5ec6b47d94d83d2c53905aa254ef84718"}
Nov 02 13:49:30.097557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:30.097533    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-qwqlh" event={"ID":"d4518e06-3103-4ae1-b027-54faab03b1a3","Type":"ContainerStarted","Data":"b18e0f6926b23ce4465698a571c0c3063244bcf67ad0bb82839b18435b6c5ff5"}
Nov 02 13:49:32.107870 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:32.107814    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-qwqlh" event={"ID":"d4518e06-3103-4ae1-b027-54faab03b1a3","Type":"ContainerStarted","Data":"5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a"}
Nov 02 13:49:32.149652 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:32.149605    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-qwqlh" podStartSLOduration=2.497663415 podStartE2EDuration="4.14958879s" totalImagesPullingTime="1.651925375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:32.148169008 +0000 UTC m=+362.899473447" watchObservedRunningTime="2026-11-02 13:49:32.14958879 +0000 UTC m=+362.900893231"
Nov 02 13:49:32.774118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:32.774061    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwqlh"]
Nov 02 13:49:33.114425 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:33.114355    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx" event={"ID":"8b0d7ea3-7cc8-4308-a764-28f01e34c5c2","Type":"ContainerStarted","Data":"2d152425eda9c2f2a75c02580309e10da330661514057b606008e82a6733b218"}
Nov 02 13:49:33.165168 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:33.165105    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx" podStartSLOduration=1.081436789 podStartE2EDuration="4.165091356s" totalImagesPullingTime="3.083654567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:49:33.164187416 +0000 UTC m=+363.915491853" watchObservedRunningTime="2026-11-02 13:49:33.165091356 +0000 UTC m=+363.916395793"
Nov 02 13:49:34.118490 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:34.118446    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-qwqlh" podUID="d4518e06-3103-4ae1-b027-54faab03b1a3" containerName="authorino" containerID="cri-o://5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a" gracePeriod=30
Nov 02 13:49:34.118974 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:34.118594    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:34.364935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:34.364910    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-qwqlh"
Nov 02 13:49:34.485504 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:34.485479    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4518e06-3103-4ae1-b027-54faab03b1a3-kube-api-access-f4l99\" (UniqueName: \"kubernetes.io/projected/d4518e06-3103-4ae1-b027-54faab03b1a3-kube-api-access-f4l99\") pod \"d4518e06-3103-4ae1-b027-54faab03b1a3\" (UID: \"d4518e06-3103-4ae1-b027-54faab03b1a3\") "
Nov 02 13:49:34.487589 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:34.487568    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4518e06-3103-4ae1-b027-54faab03b1a3-kube-api-access-f4l99" pod "d4518e06-3103-4ae1-b027-54faab03b1a3" (UID: "d4518e06-3103-4ae1-b027-54faab03b1a3"). InnerVolumeSpecName "kube-api-access-f4l99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:49:34.587006 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:34.586980    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4l99\" (UniqueName: \"kubernetes.io/projected/d4518e06-3103-4ae1-b027-54faab03b1a3-kube-api-access-f4l99\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:49:35.123524 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.123493    5034 generic.go:356] "Generic (PLEG): container finished" podID="d4518e06-3103-4ae1-b027-54faab03b1a3" containerID="5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a" exitCode=0
Nov 02 13:49:35.123940 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.123531    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-qwqlh"
Nov 02 13:49:35.123940 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.123578    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-qwqlh" event={"ID":"d4518e06-3103-4ae1-b027-54faab03b1a3","Type":"ContainerDied","Data":"5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a"}
Nov 02 13:49:35.123940 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.123617    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-qwqlh" event={"ID":"d4518e06-3103-4ae1-b027-54faab03b1a3","Type":"ContainerDied","Data":"b18e0f6926b23ce4465698a571c0c3063244bcf67ad0bb82839b18435b6c5ff5"}
Nov 02 13:49:35.123940 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.123636    5034 scope.go:122] "RemoveContainer" containerID="5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a"
Nov 02 13:49:35.133181 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.133165    5034 scope.go:122] "RemoveContainer" containerID="5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a"
Nov 02 13:49:35.133445 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:49:35.133427    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a\": container with ID starting with 5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a not found: ID does not exist" containerID="5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a"
Nov 02 13:49:35.133485 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.133455    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a"} err="failed to get container status \"5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a\": rpc error: code = NotFound desc = could not find container \"5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a\": container with ID starting with 5b609d94f75a7ca837b519ad60c0e66e3229c6857c59880925bfe1520eb5eb9a not found: ID does not exist"
Nov 02 13:49:35.175556 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.175538    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwqlh"]
Nov 02 13:49:35.180509 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.180491    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-qwqlh"]
Nov 02 13:49:35.392356 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:35.392292    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4518e06-3103-4ae1-b027-54faab03b1a3" path="/var/lib/kubelet/pods/d4518e06-3103-4ae1-b027-54faab03b1a3/volumes"
Nov 02 13:49:45.129196 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:49:45.129157    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-869dc4b47b-pbmzx"
Nov 02 13:49:48.412048 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:49:48.412014    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:50:18.412048 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:50:18.411963    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:50:28.773518 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.773482    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-gdvr6"]
Nov 02 13:50:28.773916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.773887    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d4518e06-3103-4ae1-b027-54faab03b1a3" containerName="authorino"
Nov 02 13:50:28.773916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.773899    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4518e06-3103-4ae1-b027-54faab03b1a3" containerName="authorino"
Nov 02 13:50:28.773998 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.773987    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4518e06-3103-4ae1-b027-54faab03b1a3" containerName="authorino"
Nov 02 13:50:28.777041 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.777023    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:28.798872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.798849    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-gdvr6"]
Nov 02 13:50:28.809768 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.809740    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 13:50:28.809768 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.809756    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 13:50:28.810006 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.809988    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 02 13:50:28.810006 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.810000    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-t5qv6\""
Nov 02 13:50:28.830574 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.830537    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e043e8d-d6b7-4f76-af4d-46b513dcd3f6-tls-certs\") pod \"model-serving-api-75bd89fdfc-gdvr6\" (UID: \"7e043e8d-d6b7-4f76-af4d-46b513dcd3f6\") " pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:28.830877 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.830827    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xz7r\" (UniqueName: \"kubernetes.io/projected/7e043e8d-d6b7-4f76-af4d-46b513dcd3f6-kube-api-access-5xz7r\") pod \"model-serving-api-75bd89fdfc-gdvr6\" (UID: \"7e043e8d-d6b7-4f76-af4d-46b513dcd3f6\") " pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:28.931836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.931813    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e043e8d-d6b7-4f76-af4d-46b513dcd3f6-tls-certs\") pod \"model-serving-api-75bd89fdfc-gdvr6\" (UID: \"7e043e8d-d6b7-4f76-af4d-46b513dcd3f6\") " pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:28.931978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.931849    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xz7r\" (UniqueName: \"kubernetes.io/projected/7e043e8d-d6b7-4f76-af4d-46b513dcd3f6-kube-api-access-5xz7r\") pod \"model-serving-api-75bd89fdfc-gdvr6\" (UID: \"7e043e8d-d6b7-4f76-af4d-46b513dcd3f6\") " pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:28.934235 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.934202    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e043e8d-d6b7-4f76-af4d-46b513dcd3f6-tls-certs\") pod \"model-serving-api-75bd89fdfc-gdvr6\" (UID: \"7e043e8d-d6b7-4f76-af4d-46b513dcd3f6\") " pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:28.955684 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:28.955660    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xz7r\" (UniqueName: \"kubernetes.io/projected/7e043e8d-d6b7-4f76-af4d-46b513dcd3f6-kube-api-access-5xz7r\") pod \"model-serving-api-75bd89fdfc-gdvr6\" (UID: \"7e043e8d-d6b7-4f76-af4d-46b513dcd3f6\") " pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:29.090364 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:29.090302    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:29.385071 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:29.385044    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-gdvr6"]
Nov 02 13:50:29.385797 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:50:29.385763    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e043e8d_d6b7_4f76_af4d_46b513dcd3f6.slice/crio-8f6ae545fc9b4f2f03ad8860d8de87f5b9f19e5c8a9c715135ad2dffb4094d21 WatchSource:0}: Error finding container 8f6ae545fc9b4f2f03ad8860d8de87f5b9f19e5c8a9c715135ad2dffb4094d21: Status 404 returned error can't find the container with id 8f6ae545fc9b4f2f03ad8860d8de87f5b9f19e5c8a9c715135ad2dffb4094d21
Nov 02 13:50:30.368193 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:30.368154    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6" event={"ID":"7e043e8d-d6b7-4f76-af4d-46b513dcd3f6","Type":"ContainerStarted","Data":"8f6ae545fc9b4f2f03ad8860d8de87f5b9f19e5c8a9c715135ad2dffb4094d21"}
Nov 02 13:50:31.374143 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:31.374055    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6" event={"ID":"7e043e8d-d6b7-4f76-af4d-46b513dcd3f6","Type":"ContainerStarted","Data":"3e733f7edd41c9a77f2772edbdf89aa712cc8bfd3d11e29075923eb0a64206cd"}
Nov 02 13:50:31.374498 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:31.374145    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:31.419863 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:31.419805    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6" podStartSLOduration=1.701493407 podStartE2EDuration="3.419769862s" totalImagesPullingTime="1.718276455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:50:31.41787838 +0000 UTC m=+422.169182818" watchObservedRunningTime="2026-11-02 13:50:31.419769862 +0000 UTC m=+422.171074304"
Nov 02 13:50:42.383212 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:50:42.383180    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-gdvr6"
Nov 02 13:50:48.412433 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:50:48.412394    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:51:04.705250 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.705213    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fplck"]
Nov 02 13:51:04.710386 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.710365    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fplck"
Nov 02 13:51:04.727318 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.727293    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fplck"]
Nov 02 13:51:04.738080 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.738056    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c5j78\""
Nov 02 13:51:04.744877 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.744849    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgf9d\" (UniqueName: \"kubernetes.io/projected/b977ac68-b72d-469c-8271-26243a8e8a35-kube-api-access-wgf9d\") pod \"authorino-6f6944bcdd-fplck\" (UID: \"b977ac68-b72d-469c-8271-26243a8e8a35\") " pod="kuadrant-system/authorino-6f6944bcdd-fplck"
Nov 02 13:51:04.845724 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.845684    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgf9d\" (UniqueName: \"kubernetes.io/projected/b977ac68-b72d-469c-8271-26243a8e8a35-kube-api-access-wgf9d\") pod \"authorino-6f6944bcdd-fplck\" (UID: \"b977ac68-b72d-469c-8271-26243a8e8a35\") " pod="kuadrant-system/authorino-6f6944bcdd-fplck"
Nov 02 13:51:04.873998 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.873962    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgf9d\" (UniqueName: \"kubernetes.io/projected/b977ac68-b72d-469c-8271-26243a8e8a35-kube-api-access-wgf9d\") pod \"authorino-6f6944bcdd-fplck\" (UID: \"b977ac68-b72d-469c-8271-26243a8e8a35\") " pod="kuadrant-system/authorino-6f6944bcdd-fplck"
Nov 02 13:51:04.912934 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.912892    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fplck"]
Nov 02 13:51:04.913180 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.913166    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fplck"
Nov 02 13:51:04.941463 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.941432    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8775s"]
Nov 02 13:51:04.946369 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.946345    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:04.964543 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.962629    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 13:51:04.964543 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:04.964426    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8775s"]
Nov 02 13:51:05.040136 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.040099    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8775s"]
Nov 02 13:51:05.040396 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:51:05.040373    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lhcjj tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-lhcjj tls-cert]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-8775s" podUID="e73e89b3-d39c-4369-827b-d8c0ca02a1e2"
Nov 02 13:51:05.047148 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.047117    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhcjj\" (UniqueName: \"kubernetes.io/projected/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-kube-api-access-lhcjj\") pod \"authorino-6889cbd8c4-8775s\" (UID: \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\") " pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:05.047308 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.047169    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-tls-cert\") pod \"authorino-6889cbd8c4-8775s\" (UID: \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\") " pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:05.068088 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.068053    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6498964d47-7tqdn"]
Nov 02 13:51:05.072212 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.072189    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:51:05.092043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.092013    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6498964d47-7tqdn"]
Nov 02 13:51:05.147728 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.147689    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-tls-cert\") pod \"authorino-6889cbd8c4-8775s\" (UID: \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\") " pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:05.147899 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.147821    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4bl\" (UniqueName: \"kubernetes.io/projected/655df211-d304-49bf-b474-18ef9ef0a4a0-kube-api-access-nt4bl\") pod \"authorino-6498964d47-7tqdn\" (UID: \"655df211-d304-49bf-b474-18ef9ef0a4a0\") " pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:51:05.147899 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.147879    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhcjj\" (UniqueName: \"kubernetes.io/projected/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-kube-api-access-lhcjj\") pod \"authorino-6889cbd8c4-8775s\" (UID: \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\") " pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:05.148039 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.147956    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/655df211-d304-49bf-b474-18ef9ef0a4a0-tls-cert\") pod \"authorino-6498964d47-7tqdn\" (UID: \"655df211-d304-49bf-b474-18ef9ef0a4a0\") " pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:51:05.150140 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.150116    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-tls-cert\") pod \"authorino-6889cbd8c4-8775s\" (UID: \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\") " pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:05.172171 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.172144    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhcjj\" (UniqueName: \"kubernetes.io/projected/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-kube-api-access-lhcjj\") pod \"authorino-6889cbd8c4-8775s\" (UID: \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\") " pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:05.226074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.226048    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fplck"]
Nov 02 13:51:05.227547 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:51:05.227522    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb977ac68_b72d_469c_8271_26243a8e8a35.slice/crio-f3ffdb105cb95b21441f2a5ed0fb5ad89a8ad57b94ea395dbe3a2f34bed74d6e WatchSource:0}: Error finding container f3ffdb105cb95b21441f2a5ed0fb5ad89a8ad57b94ea395dbe3a2f34bed74d6e: Status 404 returned error can't find the container with id f3ffdb105cb95b21441f2a5ed0fb5ad89a8ad57b94ea395dbe3a2f34bed74d6e
Nov 02 13:51:05.249175 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.249146    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4bl\" (UniqueName: \"kubernetes.io/projected/655df211-d304-49bf-b474-18ef9ef0a4a0-kube-api-access-nt4bl\") pod \"authorino-6498964d47-7tqdn\" (UID: \"655df211-d304-49bf-b474-18ef9ef0a4a0\") " pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:51:05.249281 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.249206    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/655df211-d304-49bf-b474-18ef9ef0a4a0-tls-cert\") pod \"authorino-6498964d47-7tqdn\" (UID: \"655df211-d304-49bf-b474-18ef9ef0a4a0\") " pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:51:05.251480 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.251458    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/655df211-d304-49bf-b474-18ef9ef0a4a0-tls-cert\") pod \"authorino-6498964d47-7tqdn\" (UID: \"655df211-d304-49bf-b474-18ef9ef0a4a0\") " pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:51:05.272840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.272819    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4bl\" (UniqueName: \"kubernetes.io/projected/655df211-d304-49bf-b474-18ef9ef0a4a0-kube-api-access-nt4bl\") pod \"authorino-6498964d47-7tqdn\" (UID: \"655df211-d304-49bf-b474-18ef9ef0a4a0\") " pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:51:05.398680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.398660    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:51:05.532417 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.532383    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fplck" event={"ID":"b977ac68-b72d-469c-8271-26243a8e8a35","Type":"ContainerStarted","Data":"f3ffdb105cb95b21441f2a5ed0fb5ad89a8ad57b94ea395dbe3a2f34bed74d6e"}
Nov 02 13:51:05.532573 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.532441    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:05.539829 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.539810    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:05.551437 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.551415    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-kube-api-access-lhcjj\" (UniqueName: \"kubernetes.io/projected/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-kube-api-access-lhcjj\") pod \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\" (UID: \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\") "
Nov 02 13:51:05.551540 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.551444    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-tls-cert\" (UniqueName: \"kubernetes.io/secret/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-tls-cert\") pod \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\" (UID: \"e73e89b3-d39c-4369-827b-d8c0ca02a1e2\") "
Nov 02 13:51:05.553565 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.553541    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-kube-api-access-lhcjj" pod "e73e89b3-d39c-4369-827b-d8c0ca02a1e2" (UID: "e73e89b3-d39c-4369-827b-d8c0ca02a1e2"). InnerVolumeSpecName "kube-api-access-lhcjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:51:05.553661 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.553556    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-tls-cert" pod "e73e89b3-d39c-4369-827b-d8c0ca02a1e2" (UID: "e73e89b3-d39c-4369-827b-d8c0ca02a1e2"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:51:05.652279 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.652254    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhcjj\" (UniqueName: \"kubernetes.io/projected/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-kube-api-access-lhcjj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:51:05.652279 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.652278    5034 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e73e89b3-d39c-4369-827b-d8c0ca02a1e2-tls-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:51:05.694654 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.694630    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6498964d47-7tqdn"]
Nov 02 13:51:05.695589 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:51:05.695565    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod655df211_d304_49bf_b474_18ef9ef0a4a0.slice/crio-39d9b9b215640a5427855901a2710f02c40627ab411cf561942de6f05a9805dd WatchSource:0}: Error finding container 39d9b9b215640a5427855901a2710f02c40627ab411cf561942de6f05a9805dd: Status 404 returned error can't find the container with id 39d9b9b215640a5427855901a2710f02c40627ab411cf561942de6f05a9805dd
Nov 02 13:51:05.748896 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.748871    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9f9xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:51:05.749284 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:05.748910    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:51:06.551062 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.550995    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fplck" event={"ID":"b977ac68-b72d-469c-8271-26243a8e8a35","Type":"ContainerStarted","Data":"43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69"}
Nov 02 13:51:06.551254 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.551074    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-fplck" podUID="b977ac68-b72d-469c-8271-26243a8e8a35" containerName="authorino" containerID="cri-o://43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69" gracePeriod=30
Nov 02 13:51:06.553481 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.553448    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6498964d47-7tqdn" event={"ID":"655df211-d304-49bf-b474-18ef9ef0a4a0","Type":"ContainerStarted","Data":"b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8"}
Nov 02 13:51:06.553611 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.553503    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6498964d47-7tqdn" event={"ID":"655df211-d304-49bf-b474-18ef9ef0a4a0","Type":"ContainerStarted","Data":"39d9b9b215640a5427855901a2710f02c40627ab411cf561942de6f05a9805dd"}
Nov 02 13:51:06.553611 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.553524    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-8775s"
Nov 02 13:51:06.592662 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.592604    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-fplck" podStartSLOduration=2.098262926 podStartE2EDuration="2.592590468s" totalImagesPullingTime="494.327542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:51:06.591688642 +0000 UTC m=+457.342993092" watchObservedRunningTime="2026-11-02 13:51:06.592590468 +0000 UTC m=+457.343894903"
Nov 02 13:51:06.660949 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.660912    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8775s"]
Nov 02 13:51:06.666385 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.666361    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-8775s"]
Nov 02 13:51:06.709587 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.709519    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6498964d47-7tqdn" podStartSLOduration=1.204375304 podStartE2EDuration="1.709496674s" totalImagesPullingTime="505.12137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:51:06.70763 +0000 UTC m=+457.458934444" watchObservedRunningTime="2026-11-02 13:51:06.709496674 +0000 UTC m=+457.460801114"
Nov 02 13:51:06.836377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.836351    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fplck"
Nov 02 13:51:06.865389 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.865355    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b977ac68-b72d-469c-8271-26243a8e8a35-kube-api-access-wgf9d\" (UniqueName: \"kubernetes.io/projected/b977ac68-b72d-469c-8271-26243a8e8a35-kube-api-access-wgf9d\") pod \"b977ac68-b72d-469c-8271-26243a8e8a35\" (UID: \"b977ac68-b72d-469c-8271-26243a8e8a35\") "
Nov 02 13:51:06.873324 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.867840    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b977ac68-b72d-469c-8271-26243a8e8a35-kube-api-access-wgf9d" pod "b977ac68-b72d-469c-8271-26243a8e8a35" (UID: "b977ac68-b72d-469c-8271-26243a8e8a35"). InnerVolumeSpecName "kube-api-access-wgf9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:51:06.966910 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:06.966884    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgf9d\" (UniqueName: \"kubernetes.io/projected/b977ac68-b72d-469c-8271-26243a8e8a35-kube-api-access-wgf9d\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:51:07.393026 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.392992    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73e89b3-d39c-4369-827b-d8c0ca02a1e2" path="/var/lib/kubelet/pods/e73e89b3-d39c-4369-827b-d8c0ca02a1e2/volumes"
Nov 02 13:51:07.559223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.559189    5034 generic.go:356] "Generic (PLEG): container finished" podID="cb1af079-7c3c-444f-a426-b716bde35270" containerID="daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e" exitCode=0
Nov 02 13:51:07.559378 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.559264    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" event={"ID":"cb1af079-7c3c-444f-a426-b716bde35270","Type":"ContainerDied","Data":"daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e"}
Nov 02 13:51:07.559672 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.559645    5034 scope.go:122] "RemoveContainer" containerID="daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e"
Nov 02 13:51:07.560677 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.560653    5034 generic.go:356] "Generic (PLEG): container finished" podID="b977ac68-b72d-469c-8271-26243a8e8a35" containerID="43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69" exitCode=0
Nov 02 13:51:07.560757 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.560721    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-fplck"
Nov 02 13:51:07.560844 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.560761    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fplck" event={"ID":"b977ac68-b72d-469c-8271-26243a8e8a35","Type":"ContainerDied","Data":"43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69"}
Nov 02 13:51:07.560844 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.560820    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-fplck" event={"ID":"b977ac68-b72d-469c-8271-26243a8e8a35","Type":"ContainerDied","Data":"f3ffdb105cb95b21441f2a5ed0fb5ad89a8ad57b94ea395dbe3a2f34bed74d6e"}
Nov 02 13:51:07.560844 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.560841    5034 scope.go:122] "RemoveContainer" containerID="43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69"
Nov 02 13:51:07.560978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.560873    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:51:07.560978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.560955    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:51:07.561068 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.561028    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:51:07.570263 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.570243    5034 scope.go:122] "RemoveContainer" containerID="43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69"
Nov 02 13:51:07.570606 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:51:07.570554    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69\": container with ID starting with 43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69 not found: ID does not exist" containerID="43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69"
Nov 02 13:51:07.570606 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.570589    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69"} err="failed to get container status \"43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69\": rpc error: code = NotFound desc = could not find container \"43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69\": container with ID starting with 43eb515513d7b095388d687148593dab621a4f02f7abf159bf20e7680218bb69 not found: ID does not exist"
Nov 02 13:51:07.661380 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.661304    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fplck"]
Nov 02 13:51:07.662748 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:07.662715    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-fplck"]
Nov 02 13:51:08.566851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:08.566818    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" event={"ID":"cb1af079-7c3c-444f-a426-b716bde35270","Type":"ContainerStarted","Data":"c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716"}
Nov 02 13:51:08.690613 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:08.690187    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:51:08.690613 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:08.690315    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9f9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:51:08.690613 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:08.690362    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 13:51:09.398385 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:09.398353    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b977ac68-b72d-469c-8271-26243a8e8a35" path="/var/lib/kubelet/pods/b977ac68-b72d-469c-8271-26243a8e8a35/volumes"
Nov 02 13:51:09.690215 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:09.690131    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9f9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:51:09.690756 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:09.690724    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 13:51:10.689214 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:10.689178    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9f9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:51:10.689497 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:10.689468    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 13:51:11.689168 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:11.689131    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9f9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:51:11.689814 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:11.689754    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 13:51:12.693935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:12.693896    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9f9xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Nov 02 13:51:12.694323 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:12.693993    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Nov 02 13:51:13.682058 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:13.682026    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:51:13.682345 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:13.682324    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:51:13.682888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:51:13.682872    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 13:51:18.411685 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:51:18.411651    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:51:48.412212 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:51:48.412132    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:52:18.411660 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:52:18.411621    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:52:40.185512 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.185479    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"]
Nov 02 13:52:40.185988 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.185920    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b977ac68-b72d-469c-8271-26243a8e8a35" containerName="authorino"
Nov 02 13:52:40.185988 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.185933    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b977ac68-b72d-469c-8271-26243a8e8a35" containerName="authorino"
Nov 02 13:52:40.186083 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.186003    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b977ac68-b72d-469c-8271-26243a8e8a35" containerName="authorino"
Nov 02 13:52:40.188203 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.188176    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.216532 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.216507    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 13:52:40.216859 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.216843    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vn5j4\""
Nov 02 13:52:40.229363 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.229343    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 13:52:40.229530 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.229517    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Nov 02 13:52:40.240505 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.240479    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"]
Nov 02 13:52:40.259183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.259162    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.259286 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.259207    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-962sd\" (UniqueName: \"kubernetes.io/projected/fb70a37a-41e1-40cd-8238-8af2875c55d6-kube-api-access-962sd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.259286 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.259236    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.259286 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.259256    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb70a37a-41e1-40cd-8238-8af2875c55d6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.259436 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.259367    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.259436 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.259388    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.360732 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.360702    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.360864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.360738    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-962sd\" (UniqueName: \"kubernetes.io/projected/fb70a37a-41e1-40cd-8238-8af2875c55d6-kube-api-access-962sd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.360864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.360765    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.360864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.360826    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb70a37a-41e1-40cd-8238-8af2875c55d6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.360864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.360850    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.361033 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.360866    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.361226 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.361195    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.361226 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.361208    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.361372 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.361303    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.363109 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.363084    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb70a37a-41e1-40cd-8238-8af2875c55d6-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.363281 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.363262    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb70a37a-41e1-40cd-8238-8af2875c55d6-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.384373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.384351    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-962sd\" (UniqueName: \"kubernetes.io/projected/fb70a37a-41e1-40cd-8238-8af2875c55d6-kube-api-access-962sd\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6fdvk\" (UID: \"fb70a37a-41e1-40cd-8238-8af2875c55d6\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.507985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.507955    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:40.809571 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.809538    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"]
Nov 02 13:52:40.809843 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:52:40.809815    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb70a37a_41e1_40cd_8238_8af2875c55d6.slice/crio-6147184fc20fa66da78fbd93834409f6a20e72b94710b6959758a2d197f8e8bf WatchSource:0}: Error finding container 6147184fc20fa66da78fbd93834409f6a20e72b94710b6959758a2d197f8e8bf: Status 404 returned error can't find the container with id 6147184fc20fa66da78fbd93834409f6a20e72b94710b6959758a2d197f8e8bf
Nov 02 13:52:40.999739 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:40.999704    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk" event={"ID":"fb70a37a-41e1-40cd-8238-8af2875c55d6","Type":"ContainerStarted","Data":"6147184fc20fa66da78fbd93834409f6a20e72b94710b6959758a2d197f8e8bf"}
Nov 02 13:52:42.644519 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.644488    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"]
Nov 02 13:52:42.648255 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.648234    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.667509 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.667480    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"]
Nov 02 13:52:42.677932 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.677901    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 13:52:42.786385 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.786351    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.786562 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.786391    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4q4\" (UniqueName: \"kubernetes.io/projected/d345d401-090c-4d81-a5fc-fbf5ca938209-kube-api-access-2m4q4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.786562 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.786420    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.786562 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.786488    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.786704 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.786607    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.786704 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.786655    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d345d401-090c-4d81-a5fc-fbf5ca938209-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.887826 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.887769    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d345d401-090c-4d81-a5fc-fbf5ca938209-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.888005 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.887871    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.888005 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.887895    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4q4\" (UniqueName: \"kubernetes.io/projected/d345d401-090c-4d81-a5fc-fbf5ca938209-kube-api-access-2m4q4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.888005 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.887944    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.888156 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.888060    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.888156 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.888111    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.888426 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.888404    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.888426 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.888419    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.888536 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.888481    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.890150 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.890127    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d345d401-090c-4d81-a5fc-fbf5ca938209-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.890323 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.890307    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d345d401-090c-4d81-a5fc-fbf5ca938209-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.916270 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.916215    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4q4\" (UniqueName: \"kubernetes.io/projected/d345d401-090c-4d81-a5fc-fbf5ca938209-kube-api-access-2m4q4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-nw8tp\" (UID: \"d345d401-090c-4d81-a5fc-fbf5ca938209\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:42.962100 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:42.962071    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:43.021193 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:43.021144    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk" event={"ID":"fb70a37a-41e1-40cd-8238-8af2875c55d6","Type":"ContainerStarted","Data":"585518bd6fd2781f31900638df483b8d4df2de7593b86a736fa17933290268d6"}
Nov 02 13:52:43.064805 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:43.064728    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk" podStartSLOduration=1.841439672 podStartE2EDuration="3.064713625s" totalImagesPullingTime="1.223273953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:43.063896904 +0000 UTC m=+553.815201344" watchObservedRunningTime="2026-11-02 13:52:43.064713625 +0000 UTC m=+553.816018063"
Nov 02 13:52:43.258681 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:43.258656    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"]
Nov 02 13:52:43.259571 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:52:43.259545    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd345d401_090c_4d81_a5fc_fbf5ca938209.slice/crio-a582ac85298c8b3fa214dfd6fd10701830a09d6030194d3d4b97df4c6fcee67c WatchSource:0}: Error finding container a582ac85298c8b3fa214dfd6fd10701830a09d6030194d3d4b97df4c6fcee67c: Status 404 returned error can't find the container with id a582ac85298c8b3fa214dfd6fd10701830a09d6030194d3d4b97df4c6fcee67c
Nov 02 13:52:43.508321 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:43.508297    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:44.028140 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:44.028097    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp" event={"ID":"d345d401-090c-4d81-a5fc-fbf5ca938209","Type":"ContainerStarted","Data":"c601916ff8b1fc7fa1d3042a6f43bdd4217251b6f8b60db984c66b9e38bc047b"}
Nov 02 13:52:44.028668 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:44.028163    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp" event={"ID":"d345d401-090c-4d81-a5fc-fbf5ca938209","Type":"ContainerStarted","Data":"a582ac85298c8b3fa214dfd6fd10701830a09d6030194d3d4b97df4c6fcee67c"}
Nov 02 13:52:44.074063 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:44.074013    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp" podStartSLOduration=1.839398466 podStartE2EDuration="2.073999551s" totalImagesPullingTime="234.601085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:44.07232379 +0000 UTC m=+554.823628229" watchObservedRunningTime="2026-11-02 13:52:44.073999551 +0000 UTC m=+554.825303988"
Nov 02 13:52:44.962241 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:44.962196    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:46.540820 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.540790    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"]
Nov 02 13:52:46.544588 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.544565    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.572877 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.572854    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 13:52:46.574649 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.574623    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"]
Nov 02 13:52:46.726510 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.726468    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.726686 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.726529    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.726686 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.726558    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.726686 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.726645    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h78tc\" (UniqueName: \"kubernetes.io/projected/e71f9623-d37b-4294-a11b-a408ecc42eb0-kube-api-access-h78tc\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.726904 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.726766    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71f9623-d37b-4294-a11b-a408ecc42eb0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.726904 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.726816    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.827764 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.827691    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71f9623-d37b-4294-a11b-a408ecc42eb0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.827764 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.827723    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.827996 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.827859    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.827996 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.827903    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.827996 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.827922    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.827996 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.827946    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h78tc\" (UniqueName: \"kubernetes.io/projected/e71f9623-d37b-4294-a11b-a408ecc42eb0-kube-api-access-h78tc\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.828333 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.828283    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.828333 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.828314    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.828519 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.828374    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.830069 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.830045    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71f9623-d37b-4294-a11b-a408ecc42eb0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.830221 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.830203    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71f9623-d37b-4294-a11b-a408ecc42eb0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.852459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.852434    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h78tc\" (UniqueName: \"kubernetes.io/projected/e71f9623-d37b-4294-a11b-a408ecc42eb0-kube-api-access-h78tc\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp\" (UID: \"e71f9623-d37b-4294-a11b-a408ecc42eb0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:46.860302 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:46.860279    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:47.186036 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:47.186010    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"]
Nov 02 13:52:47.186769 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:52:47.186738    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71f9623_d37b_4294_a11b_a408ecc42eb0.slice/crio-39ff0c0400032d992a9840bd13a68529035bb92582cc1db8a8f9612601a77543 WatchSource:0}: Error finding container 39ff0c0400032d992a9840bd13a68529035bb92582cc1db8a8f9612601a77543: Status 404 returned error can't find the container with id 39ff0c0400032d992a9840bd13a68529035bb92582cc1db8a8f9612601a77543
Nov 02 13:52:48.051882 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:48.051847    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp" event={"ID":"e71f9623-d37b-4294-a11b-a408ecc42eb0","Type":"ContainerStarted","Data":"6dc01b4e104465c7e0308fc34b5fbf258824e07a0710b75960444a6b7602e322"}
Nov 02 13:52:48.052371 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:48.051900    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp" event={"ID":"e71f9623-d37b-4294-a11b-a408ecc42eb0","Type":"ContainerStarted","Data":"39ff0c0400032d992a9840bd13a68529035bb92582cc1db8a8f9612601a77543"}
Nov 02 13:52:48.102585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:48.102529    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp" podStartSLOduration=1.8865446160000001 podStartE2EDuration="2.10251273s" totalImagesPullingTime="215.968114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:48.101025215 +0000 UTC m=+558.852329653" watchObservedRunningTime="2026-11-02 13:52:48.10251273 +0000 UTC m=+558.853817168"
Nov 02 13:52:48.429544 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:52:48.429452    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:52:48.860864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:48.860825    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:49.714085 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.714049    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"]
Nov 02 13:52:49.725056 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.725027    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.751165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.750215    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"]
Nov 02 13:52:49.754471 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.754420    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 02 13:52:49.862164 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.861589    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.862164 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.861754    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.862164 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.861814    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.862164 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.861896    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.862164 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.861926    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b539abab-f634-47a4-966f-7881e24cce86-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.862164 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.861985    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxbh\" (UniqueName: \"kubernetes.io/projected/b539abab-f634-47a4-966f-7881e24cce86-kube-api-access-2jxbh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.962871 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.962821    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.963045 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.962923    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.963045 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.962953    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.963045 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.963041    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.963209 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.963066    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b539abab-f634-47a4-966f-7881e24cce86-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.963209 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.963133    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxbh\" (UniqueName: \"kubernetes.io/projected/b539abab-f634-47a4-966f-7881e24cce86-kube-api-access-2jxbh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.963209 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.963197    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.963363 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.963294    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.963420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.963376    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.965397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.965352    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b539abab-f634-47a4-966f-7881e24cce86-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.965736 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.965713    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b539abab-f634-47a4-966f-7881e24cce86-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:49.990383 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:49.990334    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxbh\" (UniqueName: \"kubernetes.io/projected/b539abab-f634-47a4-966f-7881e24cce86-kube-api-access-2jxbh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc\" (UID: \"b539abab-f634-47a4-966f-7881e24cce86\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:50.068320 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:50.068287    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:50.392695 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:50.392670    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"]
Nov 02 13:52:50.394079 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:52:50.394054    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb539abab_f634_47a4_966f_7881e24cce86.slice/crio-4dacd941358b5b6ac191eb377b82101bfabf03d1e84e895ee1962ff0c95f90ac WatchSource:0}: Error finding container 4dacd941358b5b6ac191eb377b82101bfabf03d1e84e895ee1962ff0c95f90ac: Status 404 returned error can't find the container with id 4dacd941358b5b6ac191eb377b82101bfabf03d1e84e895ee1962ff0c95f90ac
Nov 02 13:52:50.508349 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:50.508317    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:50.520840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:50.520817    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:51.069772 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:51.069731    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" event={"ID":"b539abab-f634-47a4-966f-7881e24cce86","Type":"ContainerStarted","Data":"8c0224aaabff74e36910ec8ab3a12d89025942d16ede5f186dbfab3064196091"}
Nov 02 13:52:51.070270 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:51.069815    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" event={"ID":"b539abab-f634-47a4-966f-7881e24cce86","Type":"ContainerStarted","Data":"4dacd941358b5b6ac191eb377b82101bfabf03d1e84e895ee1962ff0c95f90ac"}
Nov 02 13:52:51.081104 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:51.081078    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6fdvk"
Nov 02 13:52:51.116350 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:51.116291    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podStartSLOduration=1.903408971 podStartE2EDuration="2.116271878s" totalImagesPullingTime="212.862907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:52:51.114806074 +0000 UTC m=+561.866110505" watchObservedRunningTime="2026-11-02 13:52:51.116271878 +0000 UTC m=+561.867576314"
Nov 02 13:52:52.069335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:52.069309    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:52:52.962916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:52.962879    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:52.976928 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:52.976903    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:53.090997 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:53.090952    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 13:52:56.861412 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:56.861381    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:56.874206 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:56.874185    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:52:57.115368 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:52:57.115277    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp"
Nov 02 13:53:00.069227 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:00.069194    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:53:00.082020 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:00.081995    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:53:00.131664 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:00.131638    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:53:11.008252 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.008172    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-67755d48d8-2qsfp"]
Nov 02 13:53:11.012340 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.012320    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67755d48d8-2qsfp"
Nov 02 13:53:11.038232 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.038208    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67755d48d8-2qsfp"]
Nov 02 13:53:11.152840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.152769    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a8b5d62-dc9c-4383-b54c-b26d25258d29-tls-cert\") pod \"authorino-67755d48d8-2qsfp\" (UID: \"0a8b5d62-dc9c-4383-b54c-b26d25258d29\") " pod="kuadrant-system/authorino-67755d48d8-2qsfp"
Nov 02 13:53:11.153003 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.152893    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxtn\" (UniqueName: \"kubernetes.io/projected/0a8b5d62-dc9c-4383-b54c-b26d25258d29-kube-api-access-vkxtn\") pod \"authorino-67755d48d8-2qsfp\" (UID: \"0a8b5d62-dc9c-4383-b54c-b26d25258d29\") " pod="kuadrant-system/authorino-67755d48d8-2qsfp"
Nov 02 13:53:11.254401 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.254367    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a8b5d62-dc9c-4383-b54c-b26d25258d29-tls-cert\") pod \"authorino-67755d48d8-2qsfp\" (UID: \"0a8b5d62-dc9c-4383-b54c-b26d25258d29\") " pod="kuadrant-system/authorino-67755d48d8-2qsfp"
Nov 02 13:53:11.254562 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.254413    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxtn\" (UniqueName: \"kubernetes.io/projected/0a8b5d62-dc9c-4383-b54c-b26d25258d29-kube-api-access-vkxtn\") pod \"authorino-67755d48d8-2qsfp\" (UID: \"0a8b5d62-dc9c-4383-b54c-b26d25258d29\") " pod="kuadrant-system/authorino-67755d48d8-2qsfp"
Nov 02 13:53:11.256882 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.256859    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0a8b5d62-dc9c-4383-b54c-b26d25258d29-tls-cert\") pod \"authorino-67755d48d8-2qsfp\" (UID: \"0a8b5d62-dc9c-4383-b54c-b26d25258d29\") " pod="kuadrant-system/authorino-67755d48d8-2qsfp"
Nov 02 13:53:11.278956 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.278897    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxtn\" (UniqueName: \"kubernetes.io/projected/0a8b5d62-dc9c-4383-b54c-b26d25258d29-kube-api-access-vkxtn\") pod \"authorino-67755d48d8-2qsfp\" (UID: \"0a8b5d62-dc9c-4383-b54c-b26d25258d29\") " pod="kuadrant-system/authorino-67755d48d8-2qsfp"
Nov 02 13:53:11.324961 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.324937    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-67755d48d8-2qsfp"
Nov 02 13:53:11.622808 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:11.622760    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a8b5d62_dc9c_4383_b54c_b26d25258d29.slice/crio-d888b9c9e99a92ec2fdade4f026749d53de3eb75c5c321e980882c879ade00c0 WatchSource:0}: Error finding container d888b9c9e99a92ec2fdade4f026749d53de3eb75c5c321e980882c879ade00c0: Status 404 returned error can't find the container with id d888b9c9e99a92ec2fdade4f026749d53de3eb75c5c321e980882c879ade00c0
Nov 02 13:53:11.623046 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:11.623015    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-67755d48d8-2qsfp"]
Nov 02 13:53:12.176392 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:12.176362    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67755d48d8-2qsfp" event={"ID":"0a8b5d62-dc9c-4383-b54c-b26d25258d29","Type":"ContainerStarted","Data":"d888b9c9e99a92ec2fdade4f026749d53de3eb75c5c321e980882c879ade00c0"}
Nov 02 13:53:13.183347 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.183313    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-67755d48d8-2qsfp" event={"ID":"0a8b5d62-dc9c-4383-b54c-b26d25258d29","Type":"ContainerStarted","Data":"8b07f8ed25d7b40416a6e756e0bd252a5599a375ce439630f66798698da60c4d"}
Nov 02 13:53:13.229935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.229878    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-67755d48d8-2qsfp" podStartSLOduration=2.7366323 podStartE2EDuration="3.229860052s" totalImagesPullingTime="493.227752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:13.228980322 +0000 UTC m=+583.980284761" watchObservedRunningTime="2026-11-02 13:53:13.229860052 +0000 UTC m=+583.981164491"
Nov 02 13:53:13.262029 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.261998    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6498964d47-7tqdn"]
Nov 02 13:53:13.262638 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.262604    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6498964d47-7tqdn" podUID="655df211-d304-49bf-b474-18ef9ef0a4a0" containerName="authorino" containerID="cri-o://b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8" gracePeriod=30
Nov 02 13:53:13.532821 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.532765    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:53:13.684202 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.684172    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/655df211-d304-49bf-b474-18ef9ef0a4a0-tls-cert\" (UniqueName: \"kubernetes.io/secret/655df211-d304-49bf-b474-18ef9ef0a4a0-tls-cert\") pod \"655df211-d304-49bf-b474-18ef9ef0a4a0\" (UID: \"655df211-d304-49bf-b474-18ef9ef0a4a0\") "
Nov 02 13:53:13.684398 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.684248    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655df211-d304-49bf-b474-18ef9ef0a4a0-kube-api-access-nt4bl\" (UniqueName: \"kubernetes.io/projected/655df211-d304-49bf-b474-18ef9ef0a4a0-kube-api-access-nt4bl\") pod \"655df211-d304-49bf-b474-18ef9ef0a4a0\" (UID: \"655df211-d304-49bf-b474-18ef9ef0a4a0\") "
Nov 02 13:53:13.686305 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.686239    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655df211-d304-49bf-b474-18ef9ef0a4a0-kube-api-access-nt4bl" pod "655df211-d304-49bf-b474-18ef9ef0a4a0" (UID: "655df211-d304-49bf-b474-18ef9ef0a4a0"). InnerVolumeSpecName "kube-api-access-nt4bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:53:13.699838 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.699810    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655df211-d304-49bf-b474-18ef9ef0a4a0-tls-cert" pod "655df211-d304-49bf-b474-18ef9ef0a4a0" (UID: "655df211-d304-49bf-b474-18ef9ef0a4a0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:53:13.785294 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.785263    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt4bl\" (UniqueName: \"kubernetes.io/projected/655df211-d304-49bf-b474-18ef9ef0a4a0-kube-api-access-nt4bl\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:53:13.785294 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:13.785296    5034 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/655df211-d304-49bf-b474-18ef9ef0a4a0-tls-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:53:14.200326 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:14.200286    5034 generic.go:356] "Generic (PLEG): container finished" podID="655df211-d304-49bf-b474-18ef9ef0a4a0" containerID="b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8" exitCode=0
Nov 02 13:53:14.200799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:14.200335    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6498964d47-7tqdn"
Nov 02 13:53:14.200799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:14.200371    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6498964d47-7tqdn" event={"ID":"655df211-d304-49bf-b474-18ef9ef0a4a0","Type":"ContainerDied","Data":"b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8"}
Nov 02 13:53:14.200799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:14.200412    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6498964d47-7tqdn" event={"ID":"655df211-d304-49bf-b474-18ef9ef0a4a0","Type":"ContainerDied","Data":"39d9b9b215640a5427855901a2710f02c40627ab411cf561942de6f05a9805dd"}
Nov 02 13:53:14.200799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:14.200434    5034 scope.go:122] "RemoveContainer" containerID="b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8"
Nov 02 13:53:14.211755 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:14.211731    5034 scope.go:122] "RemoveContainer" containerID="b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8"
Nov 02 13:53:14.212091 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:53:14.212067    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8\": container with ID starting with b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8 not found: ID does not exist" containerID="b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8"
Nov 02 13:53:14.212173 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:14.212104    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8"} err="failed to get container status \"b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8\": rpc error: code = NotFound desc = could not find container \"b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8\": container with ID starting with b9ea1a6865349473d277ff35aeebd7af5d563fcc70b4f56a779b1796404f13b8 not found: ID does not exist"
Nov 02 13:53:14.254908 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:14.254880    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6498964d47-7tqdn"]
Nov 02 13:53:14.265937 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:14.265913    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6498964d47-7tqdn"]
Nov 02 13:53:15.281800 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.281745    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"]
Nov 02 13:53:15.282242 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.282225    5034 cpu_manager.go:414] "containerMap: removing container" podUID="655df211-d304-49bf-b474-18ef9ef0a4a0" containerName="authorino"
Nov 02 13:53:15.282286 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.282244    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655df211-d304-49bf-b474-18ef9ef0a4a0" containerName="authorino"
Nov 02 13:53:15.282337 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.282328    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="655df211-d304-49bf-b474-18ef9ef0a4a0" containerName="authorino"
Nov 02 13:53:15.287344 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.286905    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.318327 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.317391    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"]
Nov 02 13:53:15.395297 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.395260    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655df211-d304-49bf-b474-18ef9ef0a4a0" path="/var/lib/kubelet/pods/655df211-d304-49bf-b474-18ef9ef0a4a0/volumes"
Nov 02 13:53:15.406744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.406715    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4x2s\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-kube-api-access-j4x2s\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.406927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.406758    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.406927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.406914    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.407047 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.406952    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.407108 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.407052    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.407166 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.407120    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.407220 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.407163    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.407308 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.407290    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.407352 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.407337    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.508532 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.508485    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.508720 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.508551    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.508720 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.508609    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.508720 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.508653    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.508925 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.508720    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4x2s\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-kube-api-access-j4x2s\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.508925 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.508748    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.508925 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.508803    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.508925 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.508829    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.508925 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.508870    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.509327 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.509296    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.510314 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.510279    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.510749 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.510720    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.510962 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.510935    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.511399 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.511376    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.512058 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.512015    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.512982 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.512952    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.533390 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.533334    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4x2s\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-kube-api-access-j4x2s\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.533554 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.533533    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-2rmgq\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.601210 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.601166    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:15.930730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.930700    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"]
Nov 02 13:53:15.931541 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:15.931517    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae84b012_0de1_42cf_a1d3_da54f50e70a9.slice/crio-0f217e206e33c6c0d3b73fffe352473e689b58ab03285f77938446e6e0203f0a WatchSource:0}: Error finding container 0f217e206e33c6c0d3b73fffe352473e689b58ab03285f77938446e6e0203f0a: Status 404 returned error can't find the container with id 0f217e206e33c6c0d3b73fffe352473e689b58ab03285f77938446e6e0203f0a
Nov 02 13:53:15.933730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.933700    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:15.933822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.933771    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:15.933867 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:15.933839    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:16.215278 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:16.215174    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" event={"ID":"ae84b012-0de1-42cf-a1d3-da54f50e70a9","Type":"ContainerStarted","Data":"2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2"}
Nov 02 13:53:16.215278 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:16.215238    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" event={"ID":"ae84b012-0de1-42cf-a1d3-da54f50e70a9","Type":"ContainerStarted","Data":"0f217e206e33c6c0d3b73fffe352473e689b58ab03285f77938446e6e0203f0a"}
Nov 02 13:53:16.285228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:16.285174    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" podStartSLOduration=1.285160139 podStartE2EDuration="1.285160139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:16.282920712 +0000 UTC m=+587.034225157" watchObservedRunningTime="2026-11-02 13:53:16.285160139 +0000 UTC m=+587.036464577"
Nov 02 13:53:16.602917 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:16.602870    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:17.603760 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:17.603721    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:53:17.604170 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:17.603812    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 13:53:18.425067 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:53:18.425017    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:53:18.601860 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:18.601821    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:53:18.602090 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:18.601885    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 13:53:19.602094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:19.602052    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:53:19.602478 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:19.602119    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 13:53:20.602176 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:20.602028    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:53:20.602176 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:20.602098    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 13:53:21.601954 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:21.601926    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:53:21.602185 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:21.602163    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 13:53:22.601960 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:22.601926    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rmgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 02 13:53:22.602364 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:22.601987    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 02 13:53:24.219166 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:24.219130    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:24.219764 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:24.219734    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:24.220602 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:24.220579    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 13:53:29.441195 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:29.441166    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 13:53:29.441657 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:29.441403    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 13:53:30.338026 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.337996    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"]
Nov 02 13:53:30.343321 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.343301    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.372744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.372719    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc21d6-0861-4473-9761-e9072aa40b46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.372893 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.372791    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.372963 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.372929    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.373028 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.372976    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.373074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.373024    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.373174 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.373156    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.373209 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.373197    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.373245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.373226    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcvqd\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-kube-api-access-xcvqd\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.373285 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.373265    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.373418 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.373404    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"]
Nov 02 13:53:30.474062 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474031    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474105    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474137    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474164    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcvqd\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-kube-api-access-xcvqd\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474217    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474276    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc21d6-0861-4473-9761-e9072aa40b46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474332    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474447    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474463    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474891 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474489    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474945 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474893    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.474945 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474928    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.475011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.474946    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.475055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.475042    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc21d6-0861-4473-9761-e9072aa40b46-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.476541 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.476512    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.476641 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.476589    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.497694 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.497674    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.497898 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.497879    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcvqd\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-kube-api-access-xcvqd\") pod \"maas-default-gateway-openshift-default-6886b96f9-6cbvw\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.656042 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.655982    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:30.967213 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.967186    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"]
Nov 02 13:53:30.967330 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:30.967244    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cc21d6_0861_4473_9761_e9072aa40b46.slice/crio-fb73912bd496642f883c4c19771cf97f1ea26c43b60c142398359cc669a2e3e0 WatchSource:0}: Error finding container fb73912bd496642f883c4c19771cf97f1ea26c43b60c142398359cc669a2e3e0: Status 404 returned error can't find the container with id fb73912bd496642f883c4c19771cf97f1ea26c43b60c142398359cc669a2e3e0
Nov 02 13:53:30.969331 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.969299    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:30.969415 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.969363    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:30.969415 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:30.969394    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:31.293442 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:31.293409    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" event={"ID":"b5cc21d6-0861-4473-9761-e9072aa40b46","Type":"ContainerStarted","Data":"8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b"}
Nov 02 13:53:31.293625 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:31.293469    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" event={"ID":"b5cc21d6-0861-4473-9761-e9072aa40b46","Type":"ContainerStarted","Data":"fb73912bd496642f883c4c19771cf97f1ea26c43b60c142398359cc669a2e3e0"}
Nov 02 13:53:31.657106 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:31.657024    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:32.657864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:32.657830    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6cbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 13:53:32.658261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:32.657888    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 13:53:33.657165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:33.657132    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6cbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 13:53:33.657335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:33.657185    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 13:53:34.656968 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:34.656937    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6cbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 13:53:34.657353 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:34.656994    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 13:53:35.656999 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:35.656961    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6cbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 13:53:35.657403 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:35.657027    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 13:53:36.656504 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:36.656468    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6cbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 13:53:36.656676 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:36.656528    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 13:53:37.657174 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:37.657138    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6cbvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 02 13:53:37.657657 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:37.657196    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 02 13:53:39.131483 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.131455    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:39.131905 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.131809    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:39.132590 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.132572    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 13:53:39.204503 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.204458    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podStartSLOduration=9.204447294 podStartE2EDuration="9.204447294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:31.33962083 +0000 UTC m=+602.090925270" watchObservedRunningTime="2026-11-02 13:53:39.204447294 +0000 UTC m=+609.955751731"
Nov 02 13:53:39.411155 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.411082    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"]
Nov 02 13:53:39.414828 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.414805    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.437053 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.437028    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c57ba6aa-openshift-default-dockercfg-tzph7\""
Nov 02 13:53:39.455591 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.455564    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"]
Nov 02 13:53:39.469441 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.469414    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpprr\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-kube-api-access-mpprr\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.469548 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.469462    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-istio-token\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.469597 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.469574    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03895243-4005-47e7-8117-6ff11632c689-istiod-ca-cert\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.469639 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.469620    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-socket\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.469742 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.469729    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-envoy\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.469797 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.469788    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-data\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.470153 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.470131    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-credential-socket\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.470261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.470244    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03895243-4005-47e7-8117-6ff11632c689-istio-podinfo\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.470306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.470291    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-certs\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.570053 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.570018    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"]
Nov 02 13:53:39.571872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.571846    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpprr\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-kube-api-access-mpprr\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.572110 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.572088    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-istio-token\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.572313 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.572294    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03895243-4005-47e7-8117-6ff11632c689-istiod-ca-cert\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.572468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.572452    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-socket\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.572604 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.572589    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-envoy\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.572767 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.572751    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-data\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.572939 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.572922    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-credential-socket\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.573088 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.573071    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03895243-4005-47e7-8117-6ff11632c689-istio-podinfo\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.573216 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.573198    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-certs\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.573840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.573732    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-data\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.573840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.572940    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-socket\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.574612 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.574585    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03895243-4005-47e7-8117-6ff11632c689-istiod-ca-cert\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.574765 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.574748    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.575007 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.574983    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-credential-socket\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.575907 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.575885    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-certs\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.576040 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.575938    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-envoy\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.577765 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.577741    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03895243-4005-47e7-8117-6ff11632c689-istio-podinfo\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.597448 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.597379    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"]
Nov 02 13:53:39.602084 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.602064    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3c148ffa-openshift-default-dockercfg-pxdms\""
Nov 02 13:53:39.611997 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.602612    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpprr\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-kube-api-access-mpprr\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.612794 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.612759    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-istio-token\") pod \"e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") " pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.674300 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.674211    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-credential-socket\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.674466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.674366    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-certs\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.674466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.674415    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa9a828-1fc0-4880-a502-56abba624eb8-istiod-ca-cert\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.674548 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.674513    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-podinfo\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.674585 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.674574    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-data\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.674621 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.674606    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-token\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.674680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.674656    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-envoy\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.674739 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.674720    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnmjc\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-kube-api-access-vnmjc\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.674813 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.674768    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-socket\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.727935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.727904    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.776342    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-certs\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.776403    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-certs\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.776468    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa9a828-1fc0-4880-a502-56abba624eb8-istiod-ca-cert\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.777907    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa9a828-1fc0-4880-a502-56abba624eb8-istiod-ca-cert\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.777958    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-podinfo\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.778036    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-data\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.778092    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-token\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.778122    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-envoy\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.778153    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnmjc\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-kube-api-access-vnmjc\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.778196    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-socket\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.778220    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-credential-socket\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.781955    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-credential-socket\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.783459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.782205    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-data\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.809050 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.787394    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-socket\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.809050 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.799397    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-envoy\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.829799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.828561    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-podinfo\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.829799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.829163    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnmjc\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-kube-api-access-vnmjc\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.830849 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.830378    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-token\") pod \"e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") " pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:39.932406 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:39.932331    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:40.086443 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.086409    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"]
Nov 02 13:53:40.086902 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:40.086876    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03895243_4005_47e7_8117_6ff11632c689.slice/crio-4c43a31e827205b5f14ebfa1c7805c431c34166d77c99e1801287d02deda4980 WatchSource:0}: Error finding container 4c43a31e827205b5f14ebfa1c7805c431c34166d77c99e1801287d02deda4980: Status 404 returned error can't find the container with id 4c43a31e827205b5f14ebfa1c7805c431c34166d77c99e1801287d02deda4980
Nov 02 13:53:40.089074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.089041    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:40.089169 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.089117    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:40.089169 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.089162    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:40.240240 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:40.240208    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa9a828_1fc0_4880_a502_56abba624eb8.slice/crio-d4beaeefde9e5d01d5ac67088651e7123a4219e0dedc4c24f0fce3d80330e5ec WatchSource:0}: Error finding container d4beaeefde9e5d01d5ac67088651e7123a4219e0dedc4c24f0fce3d80330e5ec: Status 404 returned error can't find the container with id d4beaeefde9e5d01d5ac67088651e7123a4219e0dedc4c24f0fce3d80330e5ec
Nov 02 13:53:40.242478 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.242415    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"]
Nov 02 13:53:40.243394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.243361    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:40.243480 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.243437    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:40.243528 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.243480    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:40.348651 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.348611    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" event={"ID":"4fa9a828-1fc0-4880-a502-56abba624eb8","Type":"ContainerStarted","Data":"981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8"}
Nov 02 13:53:40.348830 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.348679    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" event={"ID":"4fa9a828-1fc0-4880-a502-56abba624eb8","Type":"ContainerStarted","Data":"d4beaeefde9e5d01d5ac67088651e7123a4219e0dedc4c24f0fce3d80330e5ec"}
Nov 02 13:53:40.350822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.350796    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" event={"ID":"03895243-4005-47e7-8117-6ff11632c689","Type":"ContainerStarted","Data":"0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5"}
Nov 02 13:53:40.350976 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.350839    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" event={"ID":"03895243-4005-47e7-8117-6ff11632c689","Type":"ContainerStarted","Data":"4c43a31e827205b5f14ebfa1c7805c431c34166d77c99e1801287d02deda4980"}
Nov 02 13:53:40.399092 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.399016    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" podStartSLOduration=1.398997399 podStartE2EDuration="1.398997399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:40.397810664 +0000 UTC m=+611.149115097" watchObservedRunningTime="2026-11-02 13:53:40.398997399 +0000 UTC m=+611.150301840"
Nov 02 13:53:40.449936 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.449875    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" podStartSLOduration=1.449858409 podStartE2EDuration="1.449858409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:40.447046204 +0000 UTC m=+611.198350644" watchObservedRunningTime="2026-11-02 13:53:40.449858409 +0000 UTC m=+611.201162851"
Nov 02 13:53:40.728918 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.728845    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:40.932759 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:40.932728    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:41.733607 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:41.733582    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:41.938335 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:41.938306    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:42.366229 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:42.366200    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:42.366416 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:42.366293    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:42.366951 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:42.366930    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:53:42.367083 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:42.366981    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:53:43.769993 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.769954    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"]
Nov 02 13:53:43.773959 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.773936    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:43.800806 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.800769    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-6232c163-dockercfg-vmdrq\""
Nov 02 13:53:43.800996 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.800769    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-6232c163\""
Nov 02 13:53:43.809088 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.809066    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"]
Nov 02 13:53:43.827729 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.827700    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6232c163-5555dc89-45mr4\" (UID: \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:43.827865 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.827744    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftkjb\" (UniqueName: \"kubernetes.io/projected/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-kube-api-access-ftkjb\") pod \"payload-processing-e2e-migrate-6232c163-5555dc89-45mr4\" (UID: \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:43.928886 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.928857    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6232c163-5555dc89-45mr4\" (UID: \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:43.929015 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.928893    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftkjb\" (UniqueName: \"kubernetes.io/projected/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-kube-api-access-ftkjb\") pod \"payload-processing-e2e-migrate-6232c163-5555dc89-45mr4\" (UID: \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:43.929523 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.929505    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-plugins-config-volume\") pod \"payload-processing-e2e-migrate-6232c163-5555dc89-45mr4\" (UID: \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:43.953140 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:43.953103    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftkjb\" (UniqueName: \"kubernetes.io/projected/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-kube-api-access-ftkjb\") pod \"payload-processing-e2e-migrate-6232c163-5555dc89-45mr4\" (UID: \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\") " pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:44.090908 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:44.090808    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:44.388255 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:44.388227    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"]
Nov 02 13:53:44.389072 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:44.389047    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2711ce6a_7dc8_4b8a_b0dd_60897dcb0d99.slice/crio-6d0ac4781d2a51ecbf7de3bb2e154b5dd7d528a458358656b4e759989742d5df WatchSource:0}: Error finding container 6d0ac4781d2a51ecbf7de3bb2e154b5dd7d528a458358656b4e759989742d5df: Status 404 returned error can't find the container with id 6d0ac4781d2a51ecbf7de3bb2e154b5dd7d528a458358656b4e759989742d5df
Nov 02 13:53:44.391165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:44.391147    5034 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:53:45.385310 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.385268    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4" event={"ID":"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99","Type":"ContainerStarted","Data":"6d0ac4781d2a51ecbf7de3bb2e154b5dd7d528a458358656b4e759989742d5df"}
Nov 02 13:53:45.405289 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.405261    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"]
Nov 02 13:53:45.412069 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.410631    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:53:45.431466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.431402    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"]
Nov 02 13:53:45.440615 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.440581    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3c148ffa\""
Nov 02 13:53:45.480245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.480219    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"]
Nov 02 13:53:45.484007 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.483984    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:45.505074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.505050    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"]
Nov 02 13:53:45.505189 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.505159    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3c148ffa-dockercfg-wjtfk\""
Nov 02 13:53:45.559680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.558603    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-plugins-config-volume\") pod \"payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz\" (UID: \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\") " pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:53:45.559680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.558705    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldktz\" (UniqueName: \"kubernetes.io/projected/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-kube-api-access-ldktz\") pod \"payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw\" (UID: \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:45.559680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.558736    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt9qv\" (UniqueName: \"kubernetes.io/projected/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-kube-api-access-zt9qv\") pod \"payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz\" (UID: \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\") " pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:53:45.559680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.558791    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw\" (UID: \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:45.660767 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.660671    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-plugins-config-volume\") pod \"payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz\" (UID: \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\") " pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:53:45.660956 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.660824    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldktz\" (UniqueName: \"kubernetes.io/projected/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-kube-api-access-ldktz\") pod \"payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw\" (UID: \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:45.660956 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.660862    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt9qv\" (UniqueName: \"kubernetes.io/projected/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-kube-api-access-zt9qv\") pod \"payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz\" (UID: \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\") " pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:53:45.660956 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.660900    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw\" (UID: \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:45.661625 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.661596    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-plugins-config-volume\") pod \"payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz\" (UID: \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\") " pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:53:45.661830 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.661764    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw\" (UID: \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:45.691537 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.691508    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldktz\" (UniqueName: \"kubernetes.io/projected/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-kube-api-access-ldktz\") pod \"payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw\" (UID: \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:45.691664 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.691554    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt9qv\" (UniqueName: \"kubernetes.io/projected/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-kube-api-access-zt9qv\") pod \"payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz\" (UID: \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\") " pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:53:45.732422 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.732391    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:53:45.801878 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:45.801246    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:46.037018 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:46.036988    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fc55f1_ce04_4472_bd8d_1d3456c9513f.slice/crio-e43282941fe765c422c0c073afd739792391792a7e32bf8545bd44e279b7e8f4 WatchSource:0}: Error finding container e43282941fe765c422c0c073afd739792391792a7e32bf8545bd44e279b7e8f4: Status 404 returned error can't find the container with id e43282941fe765c422c0c073afd739792391792a7e32bf8545bd44e279b7e8f4
Nov 02 13:53:46.038256 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:46.038227    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"]
Nov 02 13:53:46.152767 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:46.152740    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"]
Nov 02 13:53:46.153080 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:46.153057    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44cf123_bcff_40c7_a69f_0ba2b86ea0de.slice/crio-75876833fbe7b4a5e2696a4fa5e4a51ed5bc2382638f55d721b0ab5a6d9e0f06 WatchSource:0}: Error finding container 75876833fbe7b4a5e2696a4fa5e4a51ed5bc2382638f55d721b0ab5a6d9e0f06: Status 404 returned error can't find the container with id 75876833fbe7b4a5e2696a4fa5e4a51ed5bc2382638f55d721b0ab5a6d9e0f06
Nov 02 13:53:46.394175 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:46.394099    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw" event={"ID":"b44cf123-bcff-40c7-a69f-0ba2b86ea0de","Type":"ContainerStarted","Data":"75876833fbe7b4a5e2696a4fa5e4a51ed5bc2382638f55d721b0ab5a6d9e0f06"}
Nov 02 13:53:46.395717 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:46.395672    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz" event={"ID":"d4fc55f1-ce04-4472-bd8d-1d3456c9513f","Type":"ContainerStarted","Data":"e43282941fe765c422c0c073afd739792391792a7e32bf8545bd44e279b7e8f4"}
Nov 02 13:53:47.407183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:47.407143    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4" event={"ID":"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99","Type":"ContainerStarted","Data":"fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2"}
Nov 02 13:53:47.408376 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:47.408284    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:47.452821 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:47.452760    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4" podStartSLOduration=1.5828241840000001 podStartE2EDuration="4.452747532s" totalImagesPullingTime="2.869923348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:47.449969898 +0000 UTC m=+618.201274337" watchObservedRunningTime="2026-11-02 13:53:47.452747532 +0000 UTC m=+618.204052033"
Nov 02 13:53:48.413256 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:53:48.413224    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:53:48.414324 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.414297    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw" event={"ID":"b44cf123-bcff-40c7-a69f-0ba2b86ea0de","Type":"ContainerStarted","Data":"2d1b1f76ff028fe6aebb966df478c815aa95ef79ccb6a1a460afa6785d96ab27"}
Nov 02 13:53:48.414450 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.414430    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:48.415743 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.415718    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz" event={"ID":"d4fc55f1-ce04-4472-bd8d-1d3456c9513f","Type":"ContainerStarted","Data":"69de04848cef34108eeb8ada57b193b345b16d47a87079d679d60c0652f67c91"}
Nov 02 13:53:48.415927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.415911    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:53:48.456323 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.456275    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw" podStartSLOduration=1.621876088 podStartE2EDuration="3.456260329s" totalImagesPullingTime="1.834384241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:48.454771011 +0000 UTC m=+619.206075449" watchObservedRunningTime="2026-11-02 13:53:48.456260329 +0000 UTC m=+619.207564767"
Nov 02 13:53:48.505637 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.505596    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz" podStartSLOduration=1.653836273 podStartE2EDuration="3.505583769s" totalImagesPullingTime="1.851747496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:48.503432501 +0000 UTC m=+619.254736939" watchObservedRunningTime="2026-11-02 13:53:48.505583769 +0000 UTC m=+619.256888233"
Nov 02 13:53:48.684162 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.684079    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"]
Nov 02 13:53:48.687751 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.687725    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.707448 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.707424    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e7818088-openshift-default-dockercfg-zbk4v\""
Nov 02 13:53:48.711003 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.710973    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"]
Nov 02 13:53:48.805536 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.805495    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-credential-socket\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.805710 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.805555    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-certs\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.805710 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.805641    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-token\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.805710 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.805674    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-socket\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.805710 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.805693    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-podinfo\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.805967 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.805883    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-envoy\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.805967 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.805927    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnbgb\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-kube-api-access-gnbgb\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.806055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.806003    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-data\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.806055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.806046    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0678ee83-5d09-4d17-919a-942ba88e5d16-istiod-ca-cert\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.907634 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.907598    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-envoy\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.907839 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.907647    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnbgb\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-kube-api-access-gnbgb\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.907839 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.907680    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-data\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.907839 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.907722    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0678ee83-5d09-4d17-919a-942ba88e5d16-istiod-ca-cert\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908018 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.907839    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-credential-socket\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908018 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.907893    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-certs\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908018 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.907935    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-token\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908163 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.908077    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-socket\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908163 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.908117    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-data\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908163 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.908135    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-podinfo\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908314 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.908229    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-credential-socket\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908488 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.908464    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-socket\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908616 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.908503    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-certs\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.908616 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.908531    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0678ee83-5d09-4d17-919a-942ba88e5d16-istiod-ca-cert\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.910037 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.910013    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-envoy\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.910474 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.910451    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-podinfo\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.934895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.934815    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnbgb\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-kube-api-access-gnbgb\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:48.936465 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:48.936435    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-token\") pod \"e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") " pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:49.004362 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:49.004332    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:49.326834 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:49.326801    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0678ee83_5d09_4d17_919a_942ba88e5d16.slice/crio-930604dcd974abe953ce0fe8630b780ca5b0147434c32a870546dbf0f71ab47f WatchSource:0}: Error finding container 930604dcd974abe953ce0fe8630b780ca5b0147434c32a870546dbf0f71ab47f: Status 404 returned error can't find the container with id 930604dcd974abe953ce0fe8630b780ca5b0147434c32a870546dbf0f71ab47f
Nov 02 13:53:49.326948 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:49.326934    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"]
Nov 02 13:53:49.329263 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:49.329229    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:49.329361 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:49.329309    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:49.329361 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:49.329354    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:53:49.423326 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:49.423286    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" event={"ID":"0678ee83-5d09-4d17-919a-942ba88e5d16","Type":"ContainerStarted","Data":"9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5"}
Nov 02 13:53:49.423742 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:49.423354    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" event={"ID":"0678ee83-5d09-4d17-919a-942ba88e5d16","Type":"ContainerStarted","Data":"930604dcd974abe953ce0fe8630b780ca5b0147434c32a870546dbf0f71ab47f"}
Nov 02 13:53:49.470072 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:49.470027    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" podStartSLOduration=1.47001298 podStartE2EDuration="1.47001298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:49.468532571 +0000 UTC m=+620.219837007" watchObservedRunningTime="2026-11-02 13:53:49.47001298 +0000 UTC m=+620.221317415"
Nov 02 13:53:50.021558 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:50.005223    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:50.028538 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:50.028509    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:50.428939 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:50.428854    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:50.429844 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:50.429825    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:53:54.051120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.051086    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"]
Nov 02 13:53:54.064341 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.064313    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:53:54.073892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.073858    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"]
Nov 02 13:53:54.097887 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.097809    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e7818088-dockercfg-xbtpm\""
Nov 02 13:53:54.110929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.110907    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e7818088\""
Nov 02 13:53:54.173525 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.173494    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba122762-3146-4b91-9ce1-2c86ebfd38e3-plugins-config-volume\") pod \"payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x\" (UID: \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\") " pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:53:54.173668 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.173547    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9nml\" (UniqueName: \"kubernetes.io/projected/ba122762-3146-4b91-9ce1-2c86ebfd38e3-kube-api-access-m9nml\") pod \"payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x\" (UID: \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\") " pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:53:54.274917 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.274873    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba122762-3146-4b91-9ce1-2c86ebfd38e3-plugins-config-volume\") pod \"payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x\" (UID: \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\") " pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:53:54.275132 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.274946    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9nml\" (UniqueName: \"kubernetes.io/projected/ba122762-3146-4b91-9ce1-2c86ebfd38e3-kube-api-access-m9nml\") pod \"payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x\" (UID: \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\") " pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:53:54.275637 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.275615    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba122762-3146-4b91-9ce1-2c86ebfd38e3-plugins-config-volume\") pod \"payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x\" (UID: \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\") " pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:53:54.319748 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.319685    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9nml\" (UniqueName: \"kubernetes.io/projected/ba122762-3146-4b91-9ce1-2c86ebfd38e3-kube-api-access-m9nml\") pod \"payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x\" (UID: \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\") " pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:53:54.394082 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.394053    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:53:54.700746 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:54.700716    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba122762_3146_4b91_9ce1_2c86ebfd38e3.slice/crio-23b8a55fe801d1c8789386b01c8607771d86e0fa037649dd9d38101fc2a4ad9b WatchSource:0}: Error finding container 23b8a55fe801d1c8789386b01c8607771d86e0fa037649dd9d38101fc2a4ad9b: Status 404 returned error can't find the container with id 23b8a55fe801d1c8789386b01c8607771d86e0fa037649dd9d38101fc2a4ad9b
Nov 02 13:53:54.701271 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:54.701248    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"]
Nov 02 13:53:55.458421 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:55.458388    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x" event={"ID":"ba122762-3146-4b91-9ce1-2c86ebfd38e3","Type":"ContainerStarted","Data":"3d55356a2173f880d06b2818b20dab378fc34d3970326610d5d576c0b2461c20"}
Nov 02 13:53:55.458852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:55.458437    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x" event={"ID":"ba122762-3146-4b91-9ce1-2c86ebfd38e3","Type":"ContainerStarted","Data":"23b8a55fe801d1c8789386b01c8607771d86e0fa037649dd9d38101fc2a4ad9b"}
Nov 02 13:53:55.458852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:55.458496    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:53:55.535301 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:55.535252    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x" podStartSLOduration=2.535235948 podStartE2EDuration="2.535235948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:53:55.534053498 +0000 UTC m=+626.285357935" watchObservedRunningTime="2026-11-02 13:53:55.535235948 +0000 UTC m=+626.286540387"
Nov 02 13:53:57.422834 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.422793    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"]
Nov 02 13:53:57.426859 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.426836    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.443377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.443356    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:53:57.456424 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.456397    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:53:57.459538 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.459509    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:53:57.459663 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.459619    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:53:57.463333 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.463308    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"]
Nov 02 13:53:57.520087 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.520051    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqnnf\" (UniqueName: \"kubernetes.io/projected/c22186e3-32fa-4e92-bc03-4b1c8c51479b-kube-api-access-xqnnf\") pod \"maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.520234 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.520107    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.520234 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.520221    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.621400 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.621361    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.621559 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.621441    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.621559 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.621515    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqnnf\" (UniqueName: \"kubernetes.io/projected/c22186e3-32fa-4e92-bc03-4b1c8c51479b-kube-api-access-xqnnf\") pod \"maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.622355 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.622322    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.623756 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.623736    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.647909 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.647880    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqnnf\" (UniqueName: \"kubernetes.io/projected/c22186e3-32fa-4e92-bc03-4b1c8c51479b-kube-api-access-xqnnf\") pod \"maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:57.740315 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:57.740274    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:53:58.077853 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:58.077669    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"]
Nov 02 13:53:58.116546 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:53:58.116064    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22186e3_32fa_4e92_bc03_4b1c8c51479b.slice/crio-842f0dcae3008ea1e5a2f86e82cfb5584695cb4f5c7162bd2bbd3437c976ccb7 WatchSource:0}: Error finding container 842f0dcae3008ea1e5a2f86e82cfb5584695cb4f5c7162bd2bbd3437c976ccb7: Status 404 returned error can't find the container with id 842f0dcae3008ea1e5a2f86e82cfb5584695cb4f5c7162bd2bbd3437c976ccb7
Nov 02 13:53:58.493806 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:58.493749    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd" event={"ID":"c22186e3-32fa-4e92-bc03-4b1c8c51479b","Type":"ContainerStarted","Data":"842f0dcae3008ea1e5a2f86e82cfb5584695cb4f5c7162bd2bbd3437c976ccb7"}
Nov 02 13:53:59.427247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:59.427222    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:53:59.427577 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:59.427559    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:53:59.427860 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:53:59.427842    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:54:04.583067 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:04.582982    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd" event={"ID":"c22186e3-32fa-4e92-bc03-4b1c8c51479b","Type":"ContainerStarted","Data":"bb132cead95c06ed4c52e460eda96a517fcd55788ffa1d8e4af34ae025adee44"}
Nov 02 13:54:04.632510 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:04.629645    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd" podStartSLOduration=2.302831453 podStartE2EDuration="7.629623239s" totalImagesPullingTime="5.326791786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:04.62866764 +0000 UTC m=+635.379972070" watchObservedRunningTime="2026-11-02 13:54:04.629623239 +0000 UTC m=+635.380927675"
Nov 02 13:54:05.209417 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.209380    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"]
Nov 02 13:54:05.213886 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.213862    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.243133 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.243103    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"]
Nov 02 13:54:05.362289 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.362242    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.362539 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.362307    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.362539 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.362504    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5d4\" (UniqueName: \"kubernetes.io/projected/945f1b6d-84bb-4c45-9f31-193671e6fe23-kube-api-access-bt5d4\") pod \"maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.463551 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.463460    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.463551 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.463518    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.463827 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.463700    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5d4\" (UniqueName: \"kubernetes.io/projected/945f1b6d-84bb-4c45-9f31-193671e6fe23-kube-api-access-bt5d4\") pod \"maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.464281 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.464258    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.466254 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.466228    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.495891 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.495832    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5d4\" (UniqueName: \"kubernetes.io/projected/945f1b6d-84bb-4c45-9f31-193671e6fe23-kube-api-access-bt5d4\") pod \"maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.528127 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.528094    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:05.630333 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.630300    5034 generic.go:356] "Generic (PLEG): container finished" podID="c22186e3-32fa-4e92-bc03-4b1c8c51479b" containerID="bb132cead95c06ed4c52e460eda96a517fcd55788ffa1d8e4af34ae025adee44" exitCode=0
Nov 02 13:54:05.630768 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.630357    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd" event={"ID":"c22186e3-32fa-4e92-bc03-4b1c8c51479b","Type":"ContainerDied","Data":"bb132cead95c06ed4c52e460eda96a517fcd55788ffa1d8e4af34ae025adee44"}
Nov 02 13:54:05.849704 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:05.849617    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"]
Nov 02 13:54:05.850075 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:05.850045    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945f1b6d_84bb_4c45_9f31_193671e6fe23.slice/crio-89b4a0fdd29695ea31ec0c6752c5c96020f0560756229cf8ded3c926ffe8795c WatchSource:0}: Error finding container 89b4a0fdd29695ea31ec0c6752c5c96020f0560756229cf8ded3c926ffe8795c: Status 404 returned error can't find the container with id 89b4a0fdd29695ea31ec0c6752c5c96020f0560756229cf8ded3c926ffe8795c
Nov 02 13:54:06.467557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:06.467468    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:54:06.637757 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:06.637709    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm" event={"ID":"945f1b6d-84bb-4c45-9f31-193671e6fe23","Type":"ContainerStarted","Data":"9d313a53151389d63f64cf97d6e5d25f6e857a7d07b3f08f471a8db2820feb1f"}
Nov 02 13:54:06.638231 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:06.637812    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm" event={"ID":"945f1b6d-84bb-4c45-9f31-193671e6fe23","Type":"ContainerStarted","Data":"89b4a0fdd29695ea31ec0c6752c5c96020f0560756229cf8ded3c926ffe8795c"}
Nov 02 13:54:06.699873 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:06.690983    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm" podStartSLOduration=1.690963593 podStartE2EDuration="1.690963593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:06.690057986 +0000 UTC m=+637.441362424" watchObservedRunningTime="2026-11-02 13:54:06.690963593 +0000 UTC m=+637.442268031"
Nov 02 13:54:06.842298 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:06.842264    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:54:07.014696 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.014669    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-openshift-ca\") pod \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") "
Nov 02 13:54:07.014850 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.014703    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c22186e3-32fa-4e92-bc03-4b1c8c51479b-kube-api-access-xqnnf\" (UniqueName: \"kubernetes.io/projected/c22186e3-32fa-4e92-bc03-4b1c8c51479b-kube-api-access-xqnnf\") pod \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") "
Nov 02 13:54:07.014931 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.014859    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-xks-ca\") pod \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\" (UID: \"c22186e3-32fa-4e92-bc03-4b1c8c51479b\") "
Nov 02 13:54:07.015148 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.015110    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-openshift-ca" pod "c22186e3-32fa-4e92-bc03-4b1c8c51479b" (UID: "c22186e3-32fa-4e92-bc03-4b1c8c51479b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:07.015349 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.015328    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:07.016997 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.016964    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c22186e3-32fa-4e92-bc03-4b1c8c51479b-kube-api-access-xqnnf" pod "c22186e3-32fa-4e92-bc03-4b1c8c51479b" (UID: "c22186e3-32fa-4e92-bc03-4b1c8c51479b"). InnerVolumeSpecName "kube-api-access-xqnnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:07.017089 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.016969    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-xks-ca" pod "c22186e3-32fa-4e92-bc03-4b1c8c51479b" (UID: "c22186e3-32fa-4e92-bc03-4b1c8c51479b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:07.118094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.116764    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqnnf\" (UniqueName: \"kubernetes.io/projected/c22186e3-32fa-4e92-bc03-4b1c8c51479b-kube-api-access-xqnnf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:07.118094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.116852    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c22186e3-32fa-4e92-bc03-4b1c8c51479b-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:07.644765 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.644695    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"
Nov 02 13:54:07.645228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.644690    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd" event={"ID":"c22186e3-32fa-4e92-bc03-4b1c8c51479b","Type":"ContainerDied","Data":"842f0dcae3008ea1e5a2f86e82cfb5584695cb4f5c7162bd2bbd3437c976ccb7"}
Nov 02 13:54:07.645228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.644843    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="842f0dcae3008ea1e5a2f86e82cfb5584695cb4f5c7162bd2bbd3437c976ccb7"
Nov 02 13:54:07.646247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.646225    5034 generic.go:356] "Generic (PLEG): container finished" podID="945f1b6d-84bb-4c45-9f31-193671e6fe23" containerID="9d313a53151389d63f64cf97d6e5d25f6e857a7d07b3f08f471a8db2820feb1f" exitCode=0
Nov 02 13:54:07.646377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.646309    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm" event={"ID":"945f1b6d-84bb-4c45-9f31-193671e6fe23","Type":"ContainerDied","Data":"9d313a53151389d63f64cf97d6e5d25f6e857a7d07b3f08f471a8db2820feb1f"}
Nov 02 13:54:07.648189 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.648171    5034 generic.go:356] "Generic (PLEG): container finished" podID="03895243-4005-47e7-8117-6ff11632c689" containerID="0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5" exitCode=0
Nov 02 13:54:07.648273 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.648228    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" event={"ID":"03895243-4005-47e7-8117-6ff11632c689","Type":"ContainerDied","Data":"0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5"}
Nov 02 13:54:07.648552 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.648539    5034 scope.go:122] "RemoveContainer" containerID="0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5"
Nov 02 13:54:07.649796 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.649749    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:07.649858 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.649829    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:07.649900 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:07.649863    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:08.304805 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.299720    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"]
Nov 02 13:54:08.306081 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.306054    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c22186e3-32fa-4e92-bc03-4b1c8c51479b" containerName="revoke-keys"
Nov 02 13:54:08.306324 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.306308    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c22186e3-32fa-4e92-bc03-4b1c8c51479b" containerName="revoke-keys"
Nov 02 13:54:08.306602 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.306590    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c22186e3-32fa-4e92-bc03-4b1c8c51479b" containerName="revoke-keys"
Nov 02 13:54:08.315001 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.314973    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.324043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.323865    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"]
Nov 02 13:54:08.431455 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.431356    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.431455 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.431453    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.431827 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.431530    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8f67\" (UniqueName: \"kubernetes.io/projected/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-kube-api-access-b8f67\") pod \"maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.533245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.533103    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.533245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.533173    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8f67\" (UniqueName: \"kubernetes.io/projected/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-kube-api-access-b8f67\") pod \"maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.533245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.533251    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.534198 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.534174    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.538808 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.536027    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.557224 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.557161    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8f67\" (UniqueName: \"kubernetes.io/projected/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-kube-api-access-b8f67\") pod \"maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.638795 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.638743    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:08.657846 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.657772    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" event={"ID":"03895243-4005-47e7-8117-6ff11632c689","Type":"ContainerStarted","Data":"077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d"}
Nov 02 13:54:08.735165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.733815    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:54:08.735165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.733990    5034 patch_prober.go:29] interesting pod/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:54:08.735165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:08.734050    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 13:54:09.036048 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.035999    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"]
Nov 02 13:54:09.067220 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.067199    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:09.240340 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.240314    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945f1b6d-84bb-4c45-9f31-193671e6fe23-kube-api-access-bt5d4\" (UniqueName: \"kubernetes.io/projected/945f1b6d-84bb-4c45-9f31-193671e6fe23-kube-api-access-bt5d4\") pod \"945f1b6d-84bb-4c45-9f31-193671e6fe23\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") "
Nov 02 13:54:09.240494 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.240450    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-openshift-ca\") pod \"945f1b6d-84bb-4c45-9f31-193671e6fe23\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") "
Nov 02 13:54:09.240874 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.240545    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-xks-ca\") pod \"945f1b6d-84bb-4c45-9f31-193671e6fe23\" (UID: \"945f1b6d-84bb-4c45-9f31-193671e6fe23\") "
Nov 02 13:54:09.241060 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.241038    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-openshift-ca" pod "945f1b6d-84bb-4c45-9f31-193671e6fe23" (UID: "945f1b6d-84bb-4c45-9f31-193671e6fe23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:09.242725 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.242696    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945f1b6d-84bb-4c45-9f31-193671e6fe23-kube-api-access-bt5d4" pod "945f1b6d-84bb-4c45-9f31-193671e6fe23" (UID: "945f1b6d-84bb-4c45-9f31-193671e6fe23"). InnerVolumeSpecName "kube-api-access-bt5d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:09.242842 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.242772    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-xks-ca" pod "945f1b6d-84bb-4c45-9f31-193671e6fe23" (UID: "945f1b6d-84bb-4c45-9f31-193671e6fe23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:09.341687 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.341651    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:09.341687 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.341681    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt5d4\" (UniqueName: \"kubernetes.io/projected/945f1b6d-84bb-4c45-9f31-193671e6fe23-kube-api-access-bt5d4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:09.341687 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.341691    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945f1b6d-84bb-4c45-9f31-193671e6fe23-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:09.664211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.664182    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"
Nov 02 13:54:09.664211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.664209    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm" event={"ID":"945f1b6d-84bb-4c45-9f31-193671e6fe23","Type":"ContainerDied","Data":"89b4a0fdd29695ea31ec0c6752c5c96020f0560756229cf8ded3c926ffe8795c"}
Nov 02 13:54:09.664792 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.664256    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89b4a0fdd29695ea31ec0c6752c5c96020f0560756229cf8ded3c926ffe8795c"
Nov 02 13:54:09.666055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.665998    5034 generic.go:356] "Generic (PLEG): container finished" podID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerID="981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8" exitCode=0
Nov 02 13:54:09.666055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.666026    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" event={"ID":"4fa9a828-1fc0-4880-a502-56abba624eb8","Type":"ContainerDied","Data":"981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8"}
Nov 02 13:54:09.666448 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.666428    5034 scope.go:122] "RemoveContainer" containerID="981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8"
Nov 02 13:54:09.667738 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.667599    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:09.667738 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.667658    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:09.667738 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.667686    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:09.668726 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.668706    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9" event={"ID":"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae","Type":"ContainerStarted","Data":"6c8079f41f8b3eb8988f623f04d03530c77007ae0e5554d83c52475b23efaa80"}
Nov 02 13:54:09.668859 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.668747    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9" event={"ID":"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae","Type":"ContainerStarted","Data":"4c28a7d78cab6b610b9e372f7f6419e9d93f862a1c3031b43102de14ef9a13b1"}
Nov 02 13:54:09.741377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.741345    5034 patch_prober.go:29] interesting pod/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:54:09.741486 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.741422    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 13:54:09.904837 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:09.891588    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9" podStartSLOduration=1.891560779 podStartE2EDuration="1.891560779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:09.890341058 +0000 UTC m=+640.641645497" watchObservedRunningTime="2026-11-02 13:54:09.891560779 +0000 UTC m=+640.642865217"
Nov 02 13:54:10.690809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:10.687269    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" event={"ID":"4fa9a828-1fc0-4880-a502-56abba624eb8","Type":"ContainerStarted","Data":"8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766"}
Nov 02 13:54:10.693305 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:10.693272    5034 generic.go:356] "Generic (PLEG): container finished" podID="5484f608-e91c-4f4f-bd57-f2b9b13ce0ae" containerID="6c8079f41f8b3eb8988f623f04d03530c77007ae0e5554d83c52475b23efaa80" exitCode=0
Nov 02 13:54:10.696803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:10.693570    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9" event={"ID":"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae","Type":"ContainerDied","Data":"6c8079f41f8b3eb8988f623f04d03530c77007ae0e5554d83c52475b23efaa80"}
Nov 02 13:54:10.730211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:10.730175    5034 patch_prober.go:29] interesting pod/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:54:10.730477 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:10.730448    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 13:54:10.952177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:10.939792    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:54:10.952177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:10.940176    5034 patch_prober.go:29] interesting pod/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 13:54:10.952177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:10.948957    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 13:54:11.729630 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:11.729316    5034 patch_prober.go:29] interesting pod/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:54:11.729630 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:11.729360    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 13:54:11.934151 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:11.934115    5034 patch_prober.go:29] interesting pod/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 13:54:11.934301 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:11.934178    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 13:54:11.963584 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:11.963555    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:12.034152 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.034125    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-openshift-ca\") pod \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") "
Nov 02 13:54:12.034344 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.034196    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-xks-ca\") pod \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") "
Nov 02 13:54:12.034344 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.034276    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-kube-api-access-b8f67\" (UniqueName: \"kubernetes.io/projected/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-kube-api-access-b8f67\") pod \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\" (UID: \"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae\") "
Nov 02 13:54:12.034530 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.034501    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-openshift-ca" pod "5484f608-e91c-4f4f-bd57-f2b9b13ce0ae" (UID: "5484f608-e91c-4f4f-bd57-f2b9b13ce0ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:12.034711 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.034692    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:12.036539 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.036517    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-kube-api-access-b8f67" pod "5484f608-e91c-4f4f-bd57-f2b9b13ce0ae" (UID: "5484f608-e91c-4f4f-bd57-f2b9b13ce0ae"). InnerVolumeSpecName "kube-api-access-b8f67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:12.036753 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.036736    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-xks-ca" pod "5484f608-e91c-4f4f-bd57-f2b9b13ce0ae" (UID: "5484f608-e91c-4f4f-bd57-f2b9b13ce0ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:12.135551 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.135516    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:12.135551 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.135549    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8f67\" (UniqueName: \"kubernetes.io/projected/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae-kube-api-access-b8f67\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:12.366165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.366131    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:54:12.366350 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.366195    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:54:12.724837 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.724745    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"
Nov 02 13:54:12.724982 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.724750    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9" event={"ID":"5484f608-e91c-4f4f-bd57-f2b9b13ce0ae","Type":"ContainerDied","Data":"4c28a7d78cab6b610b9e372f7f6419e9d93f862a1c3031b43102de14ef9a13b1"}
Nov 02 13:54:12.724982 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.724891    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c28a7d78cab6b610b9e372f7f6419e9d93f862a1c3031b43102de14ef9a13b1"
Nov 02 13:54:12.728306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.728282    5034 patch_prober.go:29] interesting pod/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:54:12.728479 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.728460    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 13:54:12.933324 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.933290    5034 patch_prober.go:29] interesting pod/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 13:54:12.933719 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:12.933342    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 13:54:13.128235 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.128197    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"]
Nov 02 13:54:13.128903 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.128886    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5484f608-e91c-4f4f-bd57-f2b9b13ce0ae" containerName="revoke-keys"
Nov 02 13:54:13.128979 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.128904    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5484f608-e91c-4f4f-bd57-f2b9b13ce0ae" containerName="revoke-keys"
Nov 02 13:54:13.128979 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.128921    5034 cpu_manager.go:414] "containerMap: removing container" podUID="945f1b6d-84bb-4c45-9f31-193671e6fe23" containerName="revoke-keys"
Nov 02 13:54:13.128979 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.128928    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945f1b6d-84bb-4c45-9f31-193671e6fe23" containerName="revoke-keys"
Nov 02 13:54:13.129136 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.129071    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5484f608-e91c-4f4f-bd57-f2b9b13ce0ae" containerName="revoke-keys"
Nov 02 13:54:13.129136 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.129084    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="945f1b6d-84bb-4c45-9f31-193671e6fe23" containerName="revoke-keys"
Nov 02 13:54:13.132737 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.132715    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.135914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.135856    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"]
Nov 02 13:54:13.146621 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.146350    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.146937 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.146912    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.147100 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.147078    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxgz\" (UniqueName: \"kubernetes.io/projected/4f3698fb-1d71-482e-b362-8d79b6b2557a-kube-api-access-vhxgz\") pod \"maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.149676 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.149654    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:54:13.149839 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.149827    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:54:13.150519 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.150496    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:54:13.150768 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.150748    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:54:13.248586 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.248550    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.248928 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.248907    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.249726 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.249703    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.249899 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.249710    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxgz\" (UniqueName: \"kubernetes.io/projected/4f3698fb-1d71-482e-b362-8d79b6b2557a-kube-api-access-vhxgz\") pod \"maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.251530 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.251504    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.292591 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.292526    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxgz\" (UniqueName: \"kubernetes.io/projected/4f3698fb-1d71-482e-b362-8d79b6b2557a-kube-api-access-vhxgz\") pod \"maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.451437 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.451346    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:13.731882 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.731756    5034 patch_prober.go:29] interesting pod/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 02 13:54:13.731882 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.731834    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 02 13:54:13.804714 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.804500    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"]
Nov 02 13:54:13.811419 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:13.811387    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3698fb_1d71_482e_b362_8d79b6b2557a.slice/crio-941831931c05115853607cd7b56e846dbdd56d9408a7ff90dbc1585cc88600c9 WatchSource:0}: Error finding container 941831931c05115853607cd7b56e846dbdd56d9408a7ff90dbc1585cc88600c9: Status 404 returned error can't find the container with id 941831931c05115853607cd7b56e846dbdd56d9408a7ff90dbc1585cc88600c9
Nov 02 13:54:13.933701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.933659    5034 patch_prober.go:29] interesting pod/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 13:54:13.934132 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:13.933722    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 13:54:14.732732 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:14.732693    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:54:14.733723 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:14.733701    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:54:14.745446 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:14.745420    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn" event={"ID":"4f3698fb-1d71-482e-b362-8d79b6b2557a","Type":"ContainerStarted","Data":"837c4ba3015ec426f06d965f1bac052f9de83e07884696de879081bc6b0643e7"}
Nov 02 13:54:14.745559 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:14.745472    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn" event={"ID":"4f3698fb-1d71-482e-b362-8d79b6b2557a","Type":"ContainerStarted","Data":"941831931c05115853607cd7b56e846dbdd56d9408a7ff90dbc1585cc88600c9"}
Nov 02 13:54:14.933411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:14.933376    5034 patch_prober.go:29] interesting pod/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 13:54:14.933573 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:14.933437    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 13:54:14.954686 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:14.954620    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn" podStartSLOduration=1.9546029310000002 podStartE2EDuration="1.954602931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:14.950110027 +0000 UTC m=+645.701414466" watchObservedRunningTime="2026-11-02 13:54:14.954602931 +0000 UTC m=+645.705907372"
Nov 02 13:54:15.756604 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:15.756562    5034 generic.go:356] "Generic (PLEG): container finished" podID="4f3698fb-1d71-482e-b362-8d79b6b2557a" containerID="837c4ba3015ec426f06d965f1bac052f9de83e07884696de879081bc6b0643e7" exitCode=0
Nov 02 13:54:15.756835 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:15.756644    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn" event={"ID":"4f3698fb-1d71-482e-b362-8d79b6b2557a","Type":"ContainerDied","Data":"837c4ba3015ec426f06d965f1bac052f9de83e07884696de879081bc6b0643e7"}
Nov 02 13:54:15.933849 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:15.933817    5034 patch_prober.go:29] interesting pod/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 02 13:54:15.935967 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:15.934014    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 02 13:54:16.250247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.250210    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"]
Nov 02 13:54:16.254959 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.254935    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.281515 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.281165    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"]
Nov 02 13:54:16.314184 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.314156    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5bcc\" (UniqueName: \"kubernetes.io/projected/c24f7d91-5b51-4207-beba-ac56e43bcc86-kube-api-access-n5bcc\") pod \"maas-api-revoke-keys-e2e-derive-e7818088-zm5rv\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.314355 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.314333    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7818088-zm5rv\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.314428 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.314398    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7818088-zm5rv\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.415426 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.415391    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7818088-zm5rv\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.415619 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.415450    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7818088-zm5rv\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.415619 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.415514    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5bcc\" (UniqueName: \"kubernetes.io/projected/c24f7d91-5b51-4207-beba-ac56e43bcc86-kube-api-access-n5bcc\") pod \"maas-api-revoke-keys-e2e-derive-e7818088-zm5rv\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.416135 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.416115    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7818088-zm5rv\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.417759 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.417730    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7818088-zm5rv\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.448177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.448150    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5bcc\" (UniqueName: \"kubernetes.io/projected/c24f7d91-5b51-4207-beba-ac56e43bcc86-kube-api-access-n5bcc\") pod \"maas-api-revoke-keys-e2e-derive-e7818088-zm5rv\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.569766 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.569686    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:16.763987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.763958    5034 generic.go:356] "Generic (PLEG): container finished" podID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerID="9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5" exitCode=0
Nov 02 13:54:16.764153 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.764022    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" event={"ID":"0678ee83-5d09-4d17-919a-942ba88e5d16","Type":"ContainerDied","Data":"9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5"}
Nov 02 13:54:16.764475 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.764461    5034 scope.go:122] "RemoveContainer" containerID="9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5"
Nov 02 13:54:16.765691 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.765665    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:16.765767 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.765727    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:16.765767 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.765761    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:16.874065 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.874016    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"]
Nov 02 13:54:16.880335 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:16.880305    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24f7d91_5b51_4207_beba_ac56e43bcc86.slice/crio-c081240e391c3a6a9df39c293e38183d0ce245def80d5eb3884d0f42b4570ab1 WatchSource:0}: Error finding container c081240e391c3a6a9df39c293e38183d0ce245def80d5eb3884d0f42b4570ab1: Status 404 returned error can't find the container with id c081240e391c3a6a9df39c293e38183d0ce245def80d5eb3884d0f42b4570ab1
Nov 02 13:54:16.913461 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.913438    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:16.937229 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.937205    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:54:16.938296 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:16.938277    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:54:17.024375 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.024344    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-openshift-ca\") pod \"4f3698fb-1d71-482e-b362-8d79b6b2557a\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") "
Nov 02 13:54:17.024550 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.024499    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-xks-ca\") pod \"4f3698fb-1d71-482e-b362-8d79b6b2557a\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") "
Nov 02 13:54:17.024550 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.024534    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3698fb-1d71-482e-b362-8d79b6b2557a-kube-api-access-vhxgz\" (UniqueName: \"kubernetes.io/projected/4f3698fb-1d71-482e-b362-8d79b6b2557a-kube-api-access-vhxgz\") pod \"4f3698fb-1d71-482e-b362-8d79b6b2557a\" (UID: \"4f3698fb-1d71-482e-b362-8d79b6b2557a\") "
Nov 02 13:54:17.025546 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.025507    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-openshift-ca" pod "4f3698fb-1d71-482e-b362-8d79b6b2557a" (UID: "4f3698fb-1d71-482e-b362-8d79b6b2557a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:17.027913 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.027880    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-xks-ca" pod "4f3698fb-1d71-482e-b362-8d79b6b2557a" (UID: "4f3698fb-1d71-482e-b362-8d79b6b2557a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:17.028059 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.028026    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3698fb-1d71-482e-b362-8d79b6b2557a-kube-api-access-vhxgz" pod "4f3698fb-1d71-482e-b362-8d79b6b2557a" (UID: "4f3698fb-1d71-482e-b362-8d79b6b2557a"). InnerVolumeSpecName "kube-api-access-vhxgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:17.125579 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.125548    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:17.125579 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.125573    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f3698fb-1d71-482e-b362-8d79b6b2557a-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:17.125579 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.125583    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxgz\" (UniqueName: \"kubernetes.io/projected/4f3698fb-1d71-482e-b362-8d79b6b2557a-kube-api-access-vhxgz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:17.797830 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.797794    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" event={"ID":"0678ee83-5d09-4d17-919a-942ba88e5d16","Type":"ContainerStarted","Data":"dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1"}
Nov 02 13:54:17.799512 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.799492    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"
Nov 02 13:54:17.799660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.799489    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn" event={"ID":"4f3698fb-1d71-482e-b362-8d79b6b2557a","Type":"ContainerDied","Data":"941831931c05115853607cd7b56e846dbdd56d9408a7ff90dbc1585cc88600c9"}
Nov 02 13:54:17.799660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.799600    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941831931c05115853607cd7b56e846dbdd56d9408a7ff90dbc1585cc88600c9"
Nov 02 13:54:17.801275 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.801195    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv" event={"ID":"c24f7d91-5b51-4207-beba-ac56e43bcc86","Type":"ContainerStarted","Data":"fd9fda86d045da505547e3fdbba86e89657406ae1604914ba050e306254f99c7"}
Nov 02 13:54:17.801275 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.801238    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv" event={"ID":"c24f7d91-5b51-4207-beba-ac56e43bcc86","Type":"ContainerStarted","Data":"c081240e391c3a6a9df39c293e38183d0ce245def80d5eb3884d0f42b4570ab1"}
Nov 02 13:54:17.895319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.894313    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"]
Nov 02 13:54:17.928379 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:17.928335    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv" podStartSLOduration=1.928320479 podStartE2EDuration="1.928320479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:17.926822372 +0000 UTC m=+648.678126801" watchObservedRunningTime="2026-11-02 13:54:17.928320479 +0000 UTC m=+648.679624927"
Nov 02 13:54:18.004904 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:18.004869    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:54:18.005583 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:18.005557    5034 patch_prober.go:29] interesting pod/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:54:18.005717 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:18.005613    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 13:54:18.420500 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:18.420459    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:54:18.690850 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:18.690095    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"]
Nov 02 13:54:18.712568 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:18.712541    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"]
Nov 02 13:54:18.811053 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:18.811015    5034 generic.go:356] "Generic (PLEG): container finished" podID="c24f7d91-5b51-4207-beba-ac56e43bcc86" containerID="fd9fda86d045da505547e3fdbba86e89657406ae1604914ba050e306254f99c7" exitCode=0
Nov 02 13:54:18.811440 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:18.811100    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv" event={"ID":"c24f7d91-5b51-4207-beba-ac56e43bcc86","Type":"ContainerDied","Data":"fd9fda86d045da505547e3fdbba86e89657406ae1604914ba050e306254f99c7"}
Nov 02 13:54:19.005263 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:19.005234    5034 patch_prober.go:29] interesting pod/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:54:19.005452 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:19.005286    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 13:54:20.005071 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.005040    5034 patch_prober.go:29] interesting pod/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:54:20.005398 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.005088    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 13:54:20.006131 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.006112    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:20.062862 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.062836    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-xks-ca\") pod \"c24f7d91-5b51-4207-beba-ac56e43bcc86\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") "
Nov 02 13:54:20.063007 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.062911    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-openshift-ca\") pod \"c24f7d91-5b51-4207-beba-ac56e43bcc86\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") "
Nov 02 13:54:20.063066 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.063013    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24f7d91-5b51-4207-beba-ac56e43bcc86-kube-api-access-n5bcc\" (UniqueName: \"kubernetes.io/projected/c24f7d91-5b51-4207-beba-ac56e43bcc86-kube-api-access-n5bcc\") pod \"c24f7d91-5b51-4207-beba-ac56e43bcc86\" (UID: \"c24f7d91-5b51-4207-beba-ac56e43bcc86\") "
Nov 02 13:54:20.063449 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.063418    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-openshift-ca" pod "c24f7d91-5b51-4207-beba-ac56e43bcc86" (UID: "c24f7d91-5b51-4207-beba-ac56e43bcc86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:20.064926 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.064902    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-xks-ca" pod "c24f7d91-5b51-4207-beba-ac56e43bcc86" (UID: "c24f7d91-5b51-4207-beba-ac56e43bcc86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:54:20.065125 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.065109    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24f7d91-5b51-4207-beba-ac56e43bcc86-kube-api-access-n5bcc" pod "c24f7d91-5b51-4207-beba-ac56e43bcc86" (UID: "c24f7d91-5b51-4207-beba-ac56e43bcc86"). InnerVolumeSpecName "kube-api-access-n5bcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:20.163861 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.163831    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5bcc\" (UniqueName: \"kubernetes.io/projected/c24f7d91-5b51-4207-beba-ac56e43bcc86-kube-api-access-n5bcc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:20.163861 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.163858    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:20.163861 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.163867    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c24f7d91-5b51-4207-beba-ac56e43bcc86-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:20.429133 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.429056    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:54:20.823233 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.823201    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"
Nov 02 13:54:20.823404 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.823235    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv" event={"ID":"c24f7d91-5b51-4207-beba-ac56e43bcc86","Type":"ContainerDied","Data":"c081240e391c3a6a9df39c293e38183d0ce245def80d5eb3884d0f42b4570ab1"}
Nov 02 13:54:20.823404 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:20.823285    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c081240e391c3a6a9df39c293e38183d0ce245def80d5eb3884d0f42b4570ab1"
Nov 02 13:54:21.004756 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:21.004728    5034 patch_prober.go:29] interesting pod/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:54:21.004916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:21.004802    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 13:54:22.005201 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:22.005165    5034 patch_prober.go:29] interesting pod/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 02 13:54:22.005721 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:22.005219    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 02 13:54:22.182627 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:22.170336    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"]
Nov 02 13:54:22.197506 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:22.197470    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ebfc3db5-v5blm"]
Nov 02 13:54:22.894699 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:22.894649    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4" podUID="2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99" containerName="payload-processing" containerID="cri-o://fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2" gracePeriod=25
Nov 02 13:54:23.010418 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.010376    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:54:23.011641 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.011622    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:54:23.164184 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.164162    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:54:23.320474 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.320431    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-kube-api-access-ftkjb\" (UniqueName: \"kubernetes.io/projected/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-kube-api-access-ftkjb\") pod \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\" (UID: \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\") "
Nov 02 13:54:23.320669 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.320494    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-plugins-config-volume\") pod \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\" (UID: \"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99\") "
Nov 02 13:54:23.320951 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.320929    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-plugins-config-volume" pod "2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99" (UID: "2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:23.322611 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.322587    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-kube-api-access-ftkjb" pod "2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99" (UID: "2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99"). InnerVolumeSpecName "kube-api-access-ftkjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:23.394233 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.394203    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945f1b6d-84bb-4c45-9f31-193671e6fe23" path="/var/lib/kubelet/pods/945f1b6d-84bb-4c45-9f31-193671e6fe23/volumes"
Nov 02 13:54:23.421928 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.421877    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftkjb\" (UniqueName: \"kubernetes.io/projected/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-kube-api-access-ftkjb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:23.421928 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.421899    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:23.690652 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.690566    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz" podUID="d4fc55f1-ce04-4472-bd8d-1d3456c9513f" containerName="payload-processing" containerID="cri-o://69de04848cef34108eeb8ada57b193b345b16d47a87079d679d60c0652f67c91" gracePeriod=25
Nov 02 13:54:23.713559 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.713511    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw" podUID="b44cf123-bcff-40c7-a69f-0ba2b86ea0de" containerName="payload-pre-processing" containerID="cri-o://2d1b1f76ff028fe6aebb966df478c815aa95ef79ccb6a1a460afa6785d96ab27" gracePeriod=25
Nov 02 13:54:23.846064 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.846032    5034 generic.go:356] "Generic (PLEG): container finished" podID="2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99" containerID="fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2" exitCode=0
Nov 02 13:54:23.846214 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.846113    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4" event={"ID":"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99","Type":"ContainerDied","Data":"fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2"}
Nov 02 13:54:23.846214 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.846117    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"
Nov 02 13:54:23.846214 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.846172    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4" event={"ID":"2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99","Type":"ContainerDied","Data":"6d0ac4781d2a51ecbf7de3bb2e154b5dd7d528a458358656b4e759989742d5df"}
Nov 02 13:54:23.846214 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.846202    5034 scope.go:122] "RemoveContainer" containerID="fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2"
Nov 02 13:54:23.848457 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.848345    5034 generic.go:356] "Generic (PLEG): container finished" podID="b44cf123-bcff-40c7-a69f-0ba2b86ea0de" containerID="2d1b1f76ff028fe6aebb966df478c815aa95ef79ccb6a1a460afa6785d96ab27" exitCode=0
Nov 02 13:54:23.849031 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.848593    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw" event={"ID":"b44cf123-bcff-40c7-a69f-0ba2b86ea0de","Type":"ContainerDied","Data":"2d1b1f76ff028fe6aebb966df478c815aa95ef79ccb6a1a460afa6785d96ab27"}
Nov 02 13:54:23.851902 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.851084    5034 generic.go:356] "Generic (PLEG): container finished" podID="d4fc55f1-ce04-4472-bd8d-1d3456c9513f" containerID="69de04848cef34108eeb8ada57b193b345b16d47a87079d679d60c0652f67c91" exitCode=0
Nov 02 13:54:23.851902 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.851859    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz" event={"ID":"d4fc55f1-ce04-4472-bd8d-1d3456c9513f","Type":"ContainerDied","Data":"69de04848cef34108eeb8ada57b193b345b16d47a87079d679d60c0652f67c91"}
Nov 02 13:54:23.863448 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.863354    5034 scope.go:122] "RemoveContainer" containerID="fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2"
Nov 02 13:54:23.863935 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:23.863873    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2\": container with ID starting with fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2 not found: ID does not exist" containerID="fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2"
Nov 02 13:54:23.863935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.863908    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2"} err="failed to get container status \"fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2\": rpc error: code = NotFound desc = could not find container \"fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2\": container with ID starting with fe7f4041e95ef9d99ae03892b621ea17275135e2075055ea3ae06f55fce81ad2 not found: ID does not exist"
Nov 02 13:54:23.915512 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.915472    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"]
Nov 02 13:54:23.917519 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:23.917498    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-6232c163-5555dc89-45mr4"]
Nov 02 13:54:24.027036 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.027005    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:54:24.053935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.053914    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.101863    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"]
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102480    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4f3698fb-1d71-482e-b362-8d79b6b2557a" containerName="revoke-keys"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102493    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3698fb-1d71-482e-b362-8d79b6b2557a" containerName="revoke-keys"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102513    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b44cf123-bcff-40c7-a69f-0ba2b86ea0de" containerName="payload-pre-processing"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102520    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44cf123-bcff-40c7-a69f-0ba2b86ea0de" containerName="payload-pre-processing"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102532    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c24f7d91-5b51-4207-beba-ac56e43bcc86" containerName="revoke-keys"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102540    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24f7d91-5b51-4207-beba-ac56e43bcc86" containerName="revoke-keys"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102554    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d4fc55f1-ce04-4472-bd8d-1d3456c9513f" containerName="payload-processing"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102561    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fc55f1-ce04-4472-bd8d-1d3456c9513f" containerName="payload-processing"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102587    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99" containerName="payload-processing"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102594    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99" containerName="payload-processing"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102713    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3698fb-1d71-482e-b362-8d79b6b2557a" containerName="revoke-keys"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102724    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44cf123-bcff-40c7-a69f-0ba2b86ea0de" containerName="payload-pre-processing"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102733    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24f7d91-5b51-4207-beba-ac56e43bcc86" containerName="revoke-keys"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102742    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fc55f1-ce04-4472-bd8d-1d3456c9513f" containerName="payload-processing"
Nov 02 13:54:24.105798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.102752    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99" containerName="payload-processing"
Nov 02 13:54:24.111112 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.111086    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.114562 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.114539    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"]
Nov 02 13:54:24.129734 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.129707    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-plugins-config-volume\") pod \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\" (UID: \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\") "
Nov 02 13:54:24.129913 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.129807    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-kube-api-access-ldktz\" (UniqueName: \"kubernetes.io/projected/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-kube-api-access-ldktz\") pod \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\" (UID: \"b44cf123-bcff-40c7-a69f-0ba2b86ea0de\") "
Nov 02 13:54:24.132523 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.131908    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1a562c-openshift-default-dockercfg-smz6x\""
Nov 02 13:54:24.132523 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.132283    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-plugins-config-volume" pod "b44cf123-bcff-40c7-a69f-0ba2b86ea0de" (UID: "b44cf123-bcff-40c7-a69f-0ba2b86ea0de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:24.146877 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.146525    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-kube-api-access-ldktz" pod "b44cf123-bcff-40c7-a69f-0ba2b86ea0de" (UID: "b44cf123-bcff-40c7-a69f-0ba2b86ea0de"). InnerVolumeSpecName "kube-api-access-ldktz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:24.230717 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.230632    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-kube-api-access-zt9qv\" (UniqueName: \"kubernetes.io/projected/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-kube-api-access-zt9qv\") pod \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\" (UID: \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\") "
Nov 02 13:54:24.230717 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.230692    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-plugins-config-volume\") pod \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\" (UID: \"d4fc55f1-ce04-4472-bd8d-1d3456c9513f\") "
Nov 02 13:54:24.231014 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.230990    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9k2f\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-kube-api-access-c9k2f\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.231075 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231042    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.231134 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231077    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-plugins-config-volume" pod "d4fc55f1-ce04-4472-bd8d-1d3456c9513f" (UID: "d4fc55f1-ce04-4472-bd8d-1d3456c9513f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:24.231134 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231114    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.231235 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231152    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.231294 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231271    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.231353 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231314    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.231353 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231345    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.231526 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231501    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.231620 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231604    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.231729 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231717    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:24.231806 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231735    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:24.231806 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.231751    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldktz\" (UniqueName: \"kubernetes.io/projected/b44cf123-bcff-40c7-a69f-0ba2b86ea0de-kube-api-access-ldktz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:24.232807 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.232764    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-kube-api-access-zt9qv" pod "d4fc55f1-ce04-4472-bd8d-1d3456c9513f" (UID: "d4fc55f1-ce04-4472-bd8d-1d3456c9513f"). InnerVolumeSpecName "kube-api-access-zt9qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344340    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344428    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344497    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9k2f\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-kube-api-access-c9k2f\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344527    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344550    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344585    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344643    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344669    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344694    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.344795    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt9qv\" (UniqueName: \"kubernetes.io/projected/d4fc55f1-ce04-4472-bd8d-1d3456c9513f-kube-api-access-zt9qv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:24.345799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.345593    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.346668 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.346445    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.346882 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.346731    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.353796 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.347126    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.357985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.356970    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.376583 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.376133    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.377359 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.377334    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.380076 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.380007    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9k2f\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-kube-api-access-c9k2f\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.402028 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.401971    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.446826 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.446644    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:24.725502 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.725470    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"]
Nov 02 13:54:24.733641 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.733614    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3c148ffa-vq8r9"]
Nov 02 13:54:24.785505 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:24.785476    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9df9c2_2c16_4bb5_bb05_ee1667367dba.slice/crio-4a1654168f433f142ad1376a24753458a70514b17c73cbb3c8ff4e15c54b8d9c WatchSource:0}: Error finding container 4a1654168f433f142ad1376a24753458a70514b17c73cbb3c8ff4e15c54b8d9c: Status 404 returned error can't find the container with id 4a1654168f433f142ad1376a24753458a70514b17c73cbb3c8ff4e15c54b8d9c
Nov 02 13:54:24.785901 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.785877    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"]
Nov 02 13:54:24.789283 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.789245    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:24.789391 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.789327    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:24.789391 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.789367    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:24.859121 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.859099    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"
Nov 02 13:54:24.859234 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.859132    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz" event={"ID":"d4fc55f1-ce04-4472-bd8d-1d3456c9513f","Type":"ContainerDied","Data":"e43282941fe765c422c0c073afd739792391792a7e32bf8545bd44e279b7e8f4"}
Nov 02 13:54:24.859234 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.859187    5034 scope.go:122] "RemoveContainer" containerID="69de04848cef34108eeb8ada57b193b345b16d47a87079d679d60c0652f67c91"
Nov 02 13:54:24.862348 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.862323    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" event={"ID":"8b9df9c2-2c16-4bb5-bb05-ee1667367dba","Type":"ContainerStarted","Data":"4a1654168f433f142ad1376a24753458a70514b17c73cbb3c8ff4e15c54b8d9c"}
Nov 02 13:54:24.863861 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.863830    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw" event={"ID":"b44cf123-bcff-40c7-a69f-0ba2b86ea0de","Type":"ContainerDied","Data":"75876833fbe7b4a5e2696a4fa5e4a51ed5bc2382638f55d721b0ab5a6d9e0f06"}
Nov 02 13:54:24.863958 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.863904    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"
Nov 02 13:54:24.873223 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.873198    5034 scope.go:122] "RemoveContainer" containerID="2d1b1f76ff028fe6aebb966df478c815aa95ef79ccb6a1a460afa6785d96ab27"
Nov 02 13:54:24.922355 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.922331    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"]
Nov 02 13:54:24.932824 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.932774    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3c148ffa-5474547dc5-4pmfw"]
Nov 02 13:54:24.970170 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.970127    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"]
Nov 02 13:54:24.973907 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:24.973883    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3c148ffa-75f7c68c-67xkz"]
Nov 02 13:54:25.031879 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.031760    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" podStartSLOduration=1.031743842 podStartE2EDuration="1.031743842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:25.030403719 +0000 UTC m=+655.781708171" watchObservedRunningTime="2026-11-02 13:54:25.031743842 +0000 UTC m=+655.783048281"
Nov 02 13:54:25.222654 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.222621    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"]
Nov 02 13:54:25.227198 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.227173    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6232c163-9jhrd"]
Nov 02 13:54:25.397626 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.397549    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99" path="/var/lib/kubelet/pods/2711ce6a-7dc8-4b8a-b0dd-60897dcb0d99/volumes"
Nov 02 13:54:25.398202 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.398177    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5484f608-e91c-4f4f-bd57-f2b9b13ce0ae" path="/var/lib/kubelet/pods/5484f608-e91c-4f4f-bd57-f2b9b13ce0ae/volumes"
Nov 02 13:54:25.398701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.398681    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b44cf123-bcff-40c7-a69f-0ba2b86ea0de" path="/var/lib/kubelet/pods/b44cf123-bcff-40c7-a69f-0ba2b86ea0de/volumes"
Nov 02 13:54:25.399691 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.399666    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c22186e3-32fa-4e92-bc03-4b1c8c51479b" path="/var/lib/kubelet/pods/c22186e3-32fa-4e92-bc03-4b1c8c51479b/volumes"
Nov 02 13:54:25.400201 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.400181    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fc55f1-ce04-4472-bd8d-1d3456c9513f" path="/var/lib/kubelet/pods/d4fc55f1-ce04-4472-bd8d-1d3456c9513f/volumes"
Nov 02 13:54:25.447769 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.447744    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:25.452687 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.452666    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:25.731192 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.731139    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"]
Nov 02 13:54:25.735813 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.735770    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c57ba6aa-wq5qn"]
Nov 02 13:54:25.875836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.875764    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" event={"ID":"8b9df9c2-2c16-4bb5-bb05-ee1667367dba","Type":"ContainerStarted","Data":"f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df"}
Nov 02 13:54:25.876876 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.876852    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:25.878233 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.878215    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:25.895683 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.895658    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"]
Nov 02 13:54:25.895913 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:25.895886    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy" containerID="cri-o://077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d" gracePeriod=30
Nov 02 13:54:26.181339 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:26.181253    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"]
Nov 02 13:54:26.181826 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:26.181556    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy" containerID="cri-o://8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766" gracePeriod=30
Nov 02 13:54:26.769055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:26.769020    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"]
Nov 02 13:54:27.160203 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.160114    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"]
Nov 02 13:54:27.167544 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.167520    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.193863 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.193842    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-0969b1-openshift-default-dockercfg-8rlbj\""
Nov 02 13:54:27.208132 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.208102    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"]
Nov 02 13:54:27.293967 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.293941    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.294132 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.293975    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.294132 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.294001    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.294211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.294151    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.294211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.294175    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzsz9\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-kube-api-access-nzsz9\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.294326 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.294306    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.294380 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.294367    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.294420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.294393    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.294420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.294415    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.395463 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.395426    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.395689 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.395474    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.395689 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.395502    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.395689 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.395587    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.395689 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.395616    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsz9\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-kube-api-access-nzsz9\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.395978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.395691    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.395978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.395739    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.395978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.395807    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.395978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.395849    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.396271 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.396246    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.396343 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.396300    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.396343 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.396323    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.396531 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.396511    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.396595 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.396581    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.396984 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.396955    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3698fb-1d71-482e-b362-8d79b6b2557a" path="/var/lib/kubelet/pods/4f3698fb-1d71-482e-b362-8d79b6b2557a/volumes"
Nov 02 13:54:27.398975 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.398949    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.399073 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.399037    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.426100 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.426048    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.435576 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.435549    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzsz9\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-kube-api-access-nzsz9\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.486434 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.486410    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:27.818437 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.818410    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"]
Nov 02 13:54:27.818892 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:27.818861    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856cd9ab_9a6f_4ba3_b8ea_87b83d868832.slice/crio-e5766821b0a005debbbae0b51f103f7bec0abfc975d678a374ca61d5fe685644 WatchSource:0}: Error finding container e5766821b0a005debbbae0b51f103f7bec0abfc975d678a374ca61d5fe685644: Status 404 returned error can't find the container with id e5766821b0a005debbbae0b51f103f7bec0abfc975d678a374ca61d5fe685644
Nov 02 13:54:27.821180 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.821145    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:27.821292 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.821227    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:27.821292 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.821272    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:27.894321 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.894293    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" event={"ID":"856cd9ab-9a6f-4ba3-b8ea-87b83d868832","Type":"ContainerStarted","Data":"357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5"}
Nov 02 13:54:27.894445 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:27.894354    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" event={"ID":"856cd9ab-9a6f-4ba3-b8ea-87b83d868832","Type":"ContainerStarted","Data":"e5766821b0a005debbbae0b51f103f7bec0abfc975d678a374ca61d5fe685644"}
Nov 02 13:54:28.486841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:28.486808    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:28.492452 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:28.492427    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:28.545043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:28.544994    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" podStartSLOduration=1.5449822869999998 podStartE2EDuration="1.544982287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:27.986993443 +0000 UTC m=+658.738297904" watchObservedRunningTime="2026-11-02 13:54:28.544982287 +0000 UTC m=+659.296286798"
Nov 02 13:54:28.899993 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:28.899916    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:28.900935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:28.900915    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:29.735129 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:29.735079    5034 patch_prober.go:29] interesting pod/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 13:54:29.735502 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:29.735151    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:54:31.176330 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.176273    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:54:31.255937 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.255898    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-kube-api-access-mpprr\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-kube-api-access-mpprr\") pod \"03895243-4005-47e7-8117-6ff11632c689\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") "
Nov 02 13:54:31.256094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.255939    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-istio-token\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-istio-token\") pod \"03895243-4005-47e7-8117-6ff11632c689\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") "
Nov 02 13:54:31.256094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.255974    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-data\") pod \"03895243-4005-47e7-8117-6ff11632c689\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") "
Nov 02 13:54:31.256094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.256040    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03895243-4005-47e7-8117-6ff11632c689-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03895243-4005-47e7-8117-6ff11632c689-istiod-ca-cert\") pod \"03895243-4005-47e7-8117-6ff11632c689\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") "
Nov 02 13:54:31.256094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.256093    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-credential-socket\") pod \"03895243-4005-47e7-8117-6ff11632c689\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") "
Nov 02 13:54:31.256314 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.256143    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-certs\") pod \"03895243-4005-47e7-8117-6ff11632c689\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") "
Nov 02 13:54:31.256314 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.256185    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-envoy\") pod \"03895243-4005-47e7-8117-6ff11632c689\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") "
Nov 02 13:54:31.256314 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.256241    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-socket\") pod \"03895243-4005-47e7-8117-6ff11632c689\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") "
Nov 02 13:54:31.256314 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.256298    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03895243-4005-47e7-8117-6ff11632c689-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03895243-4005-47e7-8117-6ff11632c689-istio-podinfo\") pod \"03895243-4005-47e7-8117-6ff11632c689\" (UID: \"03895243-4005-47e7-8117-6ff11632c689\") "
Nov 02 13:54:31.256656 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.256628    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-certs" pod "03895243-4005-47e7-8117-6ff11632c689" (UID: "03895243-4005-47e7-8117-6ff11632c689"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.257319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.256850    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-data" pod "03895243-4005-47e7-8117-6ff11632c689" (UID: "03895243-4005-47e7-8117-6ff11632c689"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.257319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.256864    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.257319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.257070    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-socket" pod "03895243-4005-47e7-8117-6ff11632c689" (UID: "03895243-4005-47e7-8117-6ff11632c689"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.257319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.257087    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-credential-socket" pod "03895243-4005-47e7-8117-6ff11632c689" (UID: "03895243-4005-47e7-8117-6ff11632c689"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.257819 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.257797    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03895243-4005-47e7-8117-6ff11632c689-istiod-ca-cert" pod "03895243-4005-47e7-8117-6ff11632c689" (UID: "03895243-4005-47e7-8117-6ff11632c689"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:31.259074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.259048    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-istio-token" pod "03895243-4005-47e7-8117-6ff11632c689" (UID: "03895243-4005-47e7-8117-6ff11632c689"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:31.259303 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.259263    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-envoy" pod "03895243-4005-47e7-8117-6ff11632c689" (UID: "03895243-4005-47e7-8117-6ff11632c689"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.259303 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.259289    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03895243-4005-47e7-8117-6ff11632c689-istio-podinfo" pod "03895243-4005-47e7-8117-6ff11632c689" (UID: "03895243-4005-47e7-8117-6ff11632c689"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:54:31.260914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.260888    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-kube-api-access-mpprr" pod "03895243-4005-47e7-8117-6ff11632c689" (UID: "03895243-4005-47e7-8117-6ff11632c689"). InnerVolumeSpecName "kube-api-access-mpprr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:31.358019 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.357974    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.358258 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.358227    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03895243-4005-47e7-8117-6ff11632c689-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.358373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.358360    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpprr\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-kube-api-access-mpprr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.358497 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.358469    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03895243-4005-47e7-8117-6ff11632c689-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.358599 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.358589    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.358691 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.358681    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03895243-4005-47e7-8117-6ff11632c689-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.358839 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.358826    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.358974 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.358960    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03895243-4005-47e7-8117-6ff11632c689-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.454242 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.454217    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:54:31.561142 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561114    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-kube-api-access-vnmjc\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-kube-api-access-vnmjc\") pod \"4fa9a828-1fc0-4880-a502-56abba624eb8\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") "
Nov 02 13:54:31.561306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561163    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-podinfo\") pod \"4fa9a828-1fc0-4880-a502-56abba624eb8\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") "
Nov 02 13:54:31.561306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561201    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-envoy\") pod \"4fa9a828-1fc0-4880-a502-56abba624eb8\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") "
Nov 02 13:54:31.561306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561221    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-data\") pod \"4fa9a828-1fc0-4880-a502-56abba624eb8\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") "
Nov 02 13:54:31.561306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561249    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fa9a828-1fc0-4880-a502-56abba624eb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa9a828-1fc0-4880-a502-56abba624eb8-istiod-ca-cert\") pod \"4fa9a828-1fc0-4880-a502-56abba624eb8\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") "
Nov 02 13:54:31.561306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561276    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-certs\") pod \"4fa9a828-1fc0-4880-a502-56abba624eb8\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") "
Nov 02 13:54:31.561587 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561388    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-socket\") pod \"4fa9a828-1fc0-4880-a502-56abba624eb8\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") "
Nov 02 13:54:31.561587 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561418    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-token\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-token\") pod \"4fa9a828-1fc0-4880-a502-56abba624eb8\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") "
Nov 02 13:54:31.561587 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561447    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-credential-socket\") pod \"4fa9a828-1fc0-4880-a502-56abba624eb8\" (UID: \"4fa9a828-1fc0-4880-a502-56abba624eb8\") "
Nov 02 13:54:31.561587 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561545    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-data" pod "4fa9a828-1fc0-4880-a502-56abba624eb8" (UID: "4fa9a828-1fc0-4880-a502-56abba624eb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.561837 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561622    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-certs" pod "4fa9a828-1fc0-4880-a502-56abba624eb8" (UID: "4fa9a828-1fc0-4880-a502-56abba624eb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.561837 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561645    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa9a828-1fc0-4880-a502-56abba624eb8-istiod-ca-cert" pod "4fa9a828-1fc0-4880-a502-56abba624eb8" (UID: "4fa9a828-1fc0-4880-a502-56abba624eb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:31.561837 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561806    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-credential-socket" pod "4fa9a828-1fc0-4880-a502-56abba624eb8" (UID: "4fa9a828-1fc0-4880-a502-56abba624eb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.561837 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.561826    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-socket" pod "4fa9a828-1fc0-4880-a502-56abba624eb8" (UID: "4fa9a828-1fc0-4880-a502-56abba624eb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.562057 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.562016    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.562057 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.562033    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.562057 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.562048    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.562217 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.562060    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fa9a828-1fc0-4880-a502-56abba624eb8-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.562217 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.562073    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.563439 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.563414    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-envoy" pod "4fa9a828-1fc0-4880-a502-56abba624eb8" (UID: "4fa9a828-1fc0-4880-a502-56abba624eb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:31.563529 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.563510    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-token" pod "4fa9a828-1fc0-4880-a502-56abba624eb8" (UID: "4fa9a828-1fc0-4880-a502-56abba624eb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:31.563713 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.563697    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-kube-api-access-vnmjc" pod "4fa9a828-1fc0-4880-a502-56abba624eb8" (UID: "4fa9a828-1fc0-4880-a502-56abba624eb8"). InnerVolumeSpecName "kube-api-access-vnmjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:31.563842 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.563824    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-podinfo" pod "4fa9a828-1fc0-4880-a502-56abba624eb8" (UID: "4fa9a828-1fc0-4880-a502-56abba624eb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:54:31.663394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.663365    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.663394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.663389    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.663394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.663398    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnmjc\" (UniqueName: \"kubernetes.io/projected/4fa9a828-1fc0-4880-a502-56abba624eb8-kube-api-access-vnmjc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.663621 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.663408    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fa9a828-1fc0-4880-a502-56abba624eb8-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:31.769318 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.769255    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x" podUID="ba122762-3146-4b91-9ce1-2c86ebfd38e3" containerName="payload-processing" containerID="cri-o://3d55356a2173f880d06b2818b20dab378fc34d3970326610d5d576c0b2461c20" gracePeriod=25
Nov 02 13:54:31.785845 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:31.785818    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba122762_3146_4b91_9ce1_2c86ebfd38e3.slice/crio-3d55356a2173f880d06b2818b20dab378fc34d3970326610d5d576c0b2461c20.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 13:54:31.922852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.922815    5034 generic.go:356] "Generic (PLEG): container finished" podID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerID="8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766" exitCode=0
Nov 02 13:54:31.923017 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.922861    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"
Nov 02 13:54:31.923017 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.922906    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" event={"ID":"4fa9a828-1fc0-4880-a502-56abba624eb8","Type":"ContainerDied","Data":"8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766"}
Nov 02 13:54:31.923017 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.922975    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w" event={"ID":"4fa9a828-1fc0-4880-a502-56abba624eb8","Type":"ContainerDied","Data":"d4beaeefde9e5d01d5ac67088651e7123a4219e0dedc4c24f0fce3d80330e5ec"}
Nov 02 13:54:31.923017 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.923009    5034 scope.go:122] "RemoveContainer" containerID="8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766"
Nov 02 13:54:31.925624 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.925596    5034 generic.go:356] "Generic (PLEG): container finished" podID="03895243-4005-47e7-8117-6ff11632c689" containerID="077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d" exitCode=0
Nov 02 13:54:31.925813 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.925687    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"
Nov 02 13:54:31.925899 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.925810    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" event={"ID":"03895243-4005-47e7-8117-6ff11632c689","Type":"ContainerDied","Data":"077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d"}
Nov 02 13:54:31.925899 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.925857    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd" event={"ID":"03895243-4005-47e7-8117-6ff11632c689","Type":"ContainerDied","Data":"4c43a31e827205b5f14ebfa1c7805c431c34166d77c99e1801287d02deda4980"}
Nov 02 13:54:31.927960 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.927925    5034 generic.go:356] "Generic (PLEG): container finished" podID="ba122762-3146-4b91-9ce1-2c86ebfd38e3" containerID="3d55356a2173f880d06b2818b20dab378fc34d3970326610d5d576c0b2461c20" exitCode=0
Nov 02 13:54:31.928061 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.927972    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x" event={"ID":"ba122762-3146-4b91-9ce1-2c86ebfd38e3","Type":"ContainerDied","Data":"3d55356a2173f880d06b2818b20dab378fc34d3970326610d5d576c0b2461c20"}
Nov 02 13:54:31.938023 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.938007    5034 scope.go:122] "RemoveContainer" containerID="981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8"
Nov 02 13:54:31.952405 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.947943    5034 scope.go:122] "RemoveContainer" containerID="8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766"
Nov 02 13:54:31.952405 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:31.949242    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766\": container with ID starting with 8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766 not found: ID does not exist" containerID="8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766"
Nov 02 13:54:31.952405 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.949316    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766"} err="failed to get container status \"8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766\": rpc error: code = NotFound desc = could not find container \"8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766\": container with ID starting with 8ef32d4dc676be2b1e9aabc7cbd573e1b5c4f6cded7ac5cd85398e18db33c766 not found: ID does not exist"
Nov 02 13:54:31.952405 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.949345    5034 scope.go:122] "RemoveContainer" containerID="981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8"
Nov 02 13:54:31.952405 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:31.950457    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8\": container with ID starting with 981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8 not found: ID does not exist" containerID="981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8"
Nov 02 13:54:31.952405 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.950487    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8"} err="failed to get container status \"981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8\": rpc error: code = NotFound desc = could not find container \"981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8\": container with ID starting with 981601100b94e33695a09f5e7b1f3d8c624eff413527700ecb0131867805b4d8 not found: ID does not exist"
Nov 02 13:54:31.952405 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.950509    5034 scope.go:122] "RemoveContainer" containerID="077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d"
Nov 02 13:54:31.970073 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.970051    5034 scope.go:122] "RemoveContainer" containerID="0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5"
Nov 02 13:54:31.981822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.981770    5034 scope.go:122] "RemoveContainer" containerID="077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d"
Nov 02 13:54:31.982161 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:31.982140    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d\": container with ID starting with 077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d not found: ID does not exist" containerID="077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d"
Nov 02 13:54:31.982252 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.982169    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d"} err="failed to get container status \"077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d\": rpc error: code = NotFound desc = could not find container \"077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d\": container with ID starting with 077bef0263ac6e49faafa9f7c3132d6ec5403c6f3ab31d4de4175eb06ec4250d not found: ID does not exist"
Nov 02 13:54:31.982252 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.982196    5034 scope.go:122] "RemoveContainer" containerID="0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5"
Nov 02 13:54:31.982495 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:31.982478    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5\": container with ID starting with 0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5 not found: ID does not exist" containerID="0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5"
Nov 02 13:54:31.982563 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:31.982511    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5"} err="failed to get container status \"0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5\": rpc error: code = NotFound desc = could not find container \"0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5\": container with ID starting with 0bf899cba91fd2139c00d72706ad42ac9562644f6f19e81795b4cfa7b2edcfd5 not found: ID does not exist"
Nov 02 13:54:32.020176 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.020120    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"]
Nov 02 13:54:32.028731 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.028712    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:54:32.032843 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.032823    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3c148ffa-openshift-default-6c6b959dfc-c2j8w"]
Nov 02 13:54:32.066466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.066444    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba122762-3146-4b91-9ce1-2c86ebfd38e3-kube-api-access-m9nml\" (UniqueName: \"kubernetes.io/projected/ba122762-3146-4b91-9ce1-2c86ebfd38e3-kube-api-access-m9nml\") pod \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\" (UID: \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\") "
Nov 02 13:54:32.066550 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.066497    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba122762-3146-4b91-9ce1-2c86ebfd38e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba122762-3146-4b91-9ce1-2c86ebfd38e3-plugins-config-volume\") pod \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\" (UID: \"ba122762-3146-4b91-9ce1-2c86ebfd38e3\") "
Nov 02 13:54:32.066958 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.066935    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba122762-3146-4b91-9ce1-2c86ebfd38e3-plugins-config-volume" pod "ba122762-3146-4b91-9ce1-2c86ebfd38e3" (UID: "ba122762-3146-4b91-9ce1-2c86ebfd38e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:32.068733 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.068712    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba122762-3146-4b91-9ce1-2c86ebfd38e3-kube-api-access-m9nml" pod "ba122762-3146-4b91-9ce1-2c86ebfd38e3" (UID: "ba122762-3146-4b91-9ce1-2c86ebfd38e3"). InnerVolumeSpecName "kube-api-access-m9nml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:32.070435 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.070407    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"]
Nov 02 13:54:32.074242 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.074219    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c57ba6aa-openshift-default-5f67788997-q6vxd"]
Nov 02 13:54:32.168423 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.168393    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9nml\" (UniqueName: \"kubernetes.io/projected/ba122762-3146-4b91-9ce1-2c86ebfd38e3-kube-api-access-m9nml\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:32.168423 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.168418    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba122762-3146-4b91-9ce1-2c86ebfd38e3-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:32.620547 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.620515    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"]
Nov 02 13:54:32.621000 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.620984    5034 cpu_manager.go:414] "containerMap: removing container" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy"
Nov 02 13:54:32.621044 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621003    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy"
Nov 02 13:54:32.621044 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621021    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy"
Nov 02 13:54:32.621044 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621026    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy"
Nov 02 13:54:32.621044 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621042    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy"
Nov 02 13:54:32.621161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621049    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy"
Nov 02 13:54:32.621161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621064    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ba122762-3146-4b91-9ce1-2c86ebfd38e3" containerName="payload-processing"
Nov 02 13:54:32.621161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621070    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba122762-3146-4b91-9ce1-2c86ebfd38e3" containerName="payload-processing"
Nov 02 13:54:32.621161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621140    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy"
Nov 02 13:54:32.621161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621151    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy"
Nov 02 13:54:32.621161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621161    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba122762-3146-4b91-9ce1-2c86ebfd38e3" containerName="payload-processing"
Nov 02 13:54:32.621332 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.621169    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" containerName="istio-proxy"
Nov 02 13:54:32.624372 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.624355    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:32.652557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.652530    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-0969b1\""
Nov 02 13:54:32.652943 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.652917    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-0969b1-dockercfg-w4rjd\""
Nov 02 13:54:32.654133 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.654107    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"]
Nov 02 13:54:32.673995 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.673970    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dbfc8e-31ad-4f65-9520-da48fb31cc62-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp\" (UID: \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:32.674116 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.674025    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hxt\" (UniqueName: \"kubernetes.io/projected/31dbfc8e-31ad-4f65-9520-da48fb31cc62-kube-api-access-82hxt\") pod \"payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp\" (UID: \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:32.721757 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.721724    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"]
Nov 02 13:54:32.722690 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.722669    5034 cpu_manager.go:414] "containerMap: removing container" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy"
Nov 02 13:54:32.722834 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.722822    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy"
Nov 02 13:54:32.723059 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.723047    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="03895243-4005-47e7-8117-6ff11632c689" containerName="istio-proxy"
Nov 02 13:54:32.730043 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.729805    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:32.747956 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.747927    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"]
Nov 02 13:54:32.775455 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.775422    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dbfc8e-31ad-4f65-9520-da48fb31cc62-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp\" (UID: \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:32.775742 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.775723    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hxt\" (UniqueName: \"kubernetes.io/projected/31dbfc8e-31ad-4f65-9520-da48fb31cc62-kube-api-access-82hxt\") pod \"payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp\" (UID: \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:32.775912 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.775895    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721fe7a3-b75c-4796-9efb-6dac45991d50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm\" (UID: \"721fe7a3-b75c-4796-9efb-6dac45991d50\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:32.776079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.776064    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbzbg\" (UniqueName: \"kubernetes.io/projected/721fe7a3-b75c-4796-9efb-6dac45991d50-kube-api-access-cbzbg\") pod \"payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm\" (UID: \"721fe7a3-b75c-4796-9efb-6dac45991d50\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:32.777342 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.777313    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dbfc8e-31ad-4f65-9520-da48fb31cc62-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp\" (UID: \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:32.817999 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.817968    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hxt\" (UniqueName: \"kubernetes.io/projected/31dbfc8e-31ad-4f65-9520-da48fb31cc62-kube-api-access-82hxt\") pod \"payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp\" (UID: \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:32.881487 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.881383    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721fe7a3-b75c-4796-9efb-6dac45991d50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm\" (UID: \"721fe7a3-b75c-4796-9efb-6dac45991d50\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:32.881659 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.881495    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbzbg\" (UniqueName: \"kubernetes.io/projected/721fe7a3-b75c-4796-9efb-6dac45991d50-kube-api-access-cbzbg\") pod \"payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm\" (UID: \"721fe7a3-b75c-4796-9efb-6dac45991d50\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:32.882895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.882748    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721fe7a3-b75c-4796-9efb-6dac45991d50-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm\" (UID: \"721fe7a3-b75c-4796-9efb-6dac45991d50\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:32.906932 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.906907    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbzbg\" (UniqueName: \"kubernetes.io/projected/721fe7a3-b75c-4796-9efb-6dac45991d50-kube-api-access-cbzbg\") pod \"payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm\" (UID: \"721fe7a3-b75c-4796-9efb-6dac45991d50\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:32.936827 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.936800    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"
Nov 02 13:54:32.936827 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.936813    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x" event={"ID":"ba122762-3146-4b91-9ce1-2c86ebfd38e3","Type":"ContainerDied","Data":"23b8a55fe801d1c8789386b01c8607771d86e0fa037649dd9d38101fc2a4ad9b"}
Nov 02 13:54:32.937059 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.936876    5034 scope.go:122] "RemoveContainer" containerID="3d55356a2173f880d06b2818b20dab378fc34d3970326610d5d576c0b2461c20"
Nov 02 13:54:32.938310 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:32.938185    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:33.022141 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.014163    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"]
Nov 02 13:54:33.022141 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.020511    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"]
Nov 02 13:54:33.024152 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.024049    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e7818088-85c687b7d9-qt58x"]
Nov 02 13:54:33.028822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.028196    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7818088-zm5rv"]
Nov 02 13:54:33.076934 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.076905    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:33.161817 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.161792    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"]
Nov 02 13:54:33.162606 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.162559    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy" containerID="cri-o://dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1" gracePeriod=30
Nov 02 13:54:33.274829 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.254216    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"]
Nov 02 13:54:33.278453 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:33.276334    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dbfc8e_31ad_4f65_9520_da48fb31cc62.slice/crio-12e5385d11aaf9d6f65cfb4bf6236de48861e85e57b7c39e80089d1684be036d WatchSource:0}: Error finding container 12e5385d11aaf9d6f65cfb4bf6236de48861e85e57b7c39e80089d1684be036d: Status 404 returned error can't find the container with id 12e5385d11aaf9d6f65cfb4bf6236de48861e85e57b7c39e80089d1684be036d
Nov 02 13:54:33.393692 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.393632    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03895243-4005-47e7-8117-6ff11632c689" path="/var/lib/kubelet/pods/03895243-4005-47e7-8117-6ff11632c689/volumes"
Nov 02 13:54:33.394094 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.394072    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fa9a828-1fc0-4880-a502-56abba624eb8" path="/var/lib/kubelet/pods/4fa9a828-1fc0-4880-a502-56abba624eb8/volumes"
Nov 02 13:54:33.394601 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.394580    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba122762-3146-4b91-9ce1-2c86ebfd38e3" path="/var/lib/kubelet/pods/ba122762-3146-4b91-9ce1-2c86ebfd38e3/volumes"
Nov 02 13:54:33.395618 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.395595    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24f7d91-5b51-4207-beba-ac56e43bcc86" path="/var/lib/kubelet/pods/c24f7d91-5b51-4207-beba-ac56e43bcc86/volumes"
Nov 02 13:54:33.397210 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.397184    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"]
Nov 02 13:54:33.397610 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:33.397590    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721fe7a3_b75c_4796_9efb_6dac45991d50.slice/crio-d18d5919c27b6297e9b5a58b7b9401c73caddaf289bcd423929ceb47592d8d60 WatchSource:0}: Error finding container d18d5919c27b6297e9b5a58b7b9401c73caddaf289bcd423929ceb47592d8d60: Status 404 returned error can't find the container with id d18d5919c27b6297e9b5a58b7b9401c73caddaf289bcd423929ceb47592d8d60
Nov 02 13:54:33.945706 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.945670    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm" event={"ID":"721fe7a3-b75c-4796-9efb-6dac45991d50","Type":"ContainerStarted","Data":"7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d"}
Nov 02 13:54:33.946544 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.945730    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm" event={"ID":"721fe7a3-b75c-4796-9efb-6dac45991d50","Type":"ContainerStarted","Data":"d18d5919c27b6297e9b5a58b7b9401c73caddaf289bcd423929ceb47592d8d60"}
Nov 02 13:54:33.946544 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.945906    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:33.947275 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.947253    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp" event={"ID":"31dbfc8e-31ad-4f65-9520-da48fb31cc62","Type":"ContainerStarted","Data":"b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725"}
Nov 02 13:54:33.947369 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.947302    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp" event={"ID":"31dbfc8e-31ad-4f65-9520-da48fb31cc62","Type":"ContainerStarted","Data":"12e5385d11aaf9d6f65cfb4bf6236de48861e85e57b7c39e80089d1684be036d"}
Nov 02 13:54:33.947369 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.947331    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:33.994191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:33.994132    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm" podStartSLOduration=1.99411504 podStartE2EDuration="1.99411504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:33.993270068 +0000 UTC m=+664.744574506" watchObservedRunningTime="2026-11-02 13:54:33.99411504 +0000 UTC m=+664.745419479"
Nov 02 13:54:34.042055 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.042011    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp" podStartSLOduration=2.0419994949999998 podStartE2EDuration="2.041999495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:34.040938521 +0000 UTC m=+664.792242960" watchObservedRunningTime="2026-11-02 13:54:34.041999495 +0000 UTC m=+664.793303932"
Nov 02 13:54:34.409551 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.409513    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"]
Nov 02 13:54:34.413577 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.413554    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.430155 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.430121    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-92f249-openshift-default-dockercfg-p2t2n\""
Nov 02 13:54:34.468559 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.468528    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"]
Nov 02 13:54:34.503979 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.503943    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.504135 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.504059    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7kr6\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-kube-api-access-n7kr6\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.504218 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.504192    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.504362 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.504250    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.504516 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.504498    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.504602 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.504523    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.504602 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.504575    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.504602 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.504598    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.504795 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.504721    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605315 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605278    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605315 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605318    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605347    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605384    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605440    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605470    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605505    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7kr6\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-kube-api-access-n7kr6\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605563    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605613    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605732    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605817    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.605875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605865    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.606166 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.605958    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.606166 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.606125    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.608343 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.608315    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.610807 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.608606    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.634293 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.634258    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7kr6\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-kube-api-access-n7kr6\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.635039 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.635013    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:34.731256 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:34.731225    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:35.071443 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.071410    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"]
Nov 02 13:54:35.076966 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:35.076935    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91aab7bc_5ef8_4037_a001_3fefd605f6b4.slice/crio-afa0774e1077e07a32afded7fb2f762fedb37372fae829f2c7f24511e41112f7 WatchSource:0}: Error finding container afa0774e1077e07a32afded7fb2f762fedb37372fae829f2c7f24511e41112f7: Status 404 returned error can't find the container with id afa0774e1077e07a32afded7fb2f762fedb37372fae829f2c7f24511e41112f7
Nov 02 13:54:35.082730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.082062    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:35.082730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.082147    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:35.082730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.082187    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:35.631909 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.631880    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"]
Nov 02 13:54:35.655822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.654171    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:35.669836 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.668440    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"]
Nov 02 13:54:35.721983 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.721940    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c619c67a-04e0-45a2-8759-4f278b5b3326-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z\" (UID: \"c619c67a-04e0-45a2-8759-4f278b5b3326\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:35.722136 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.722003    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnv4h\" (UniqueName: \"kubernetes.io/projected/c619c67a-04e0-45a2-8759-4f278b5b3326-kube-api-access-mnv4h\") pod \"payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z\" (UID: \"c619c67a-04e0-45a2-8759-4f278b5b3326\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:35.766669 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.766645    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f30173\""
Nov 02 13:54:35.767171 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.767155    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f30173-dockercfg-b26ww\""
Nov 02 13:54:35.823558 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.823530    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c619c67a-04e0-45a2-8759-4f278b5b3326-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z\" (UID: \"c619c67a-04e0-45a2-8759-4f278b5b3326\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:35.823671 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.823567    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnv4h\" (UniqueName: \"kubernetes.io/projected/c619c67a-04e0-45a2-8759-4f278b5b3326-kube-api-access-mnv4h\") pod \"payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z\" (UID: \"c619c67a-04e0-45a2-8759-4f278b5b3326\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:35.824216 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.824196    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c619c67a-04e0-45a2-8759-4f278b5b3326-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z\" (UID: \"c619c67a-04e0-45a2-8759-4f278b5b3326\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:35.850333 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.850304    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnv4h\" (UniqueName: \"kubernetes.io/projected/c619c67a-04e0-45a2-8759-4f278b5b3326-kube-api-access-mnv4h\") pod \"payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z\" (UID: \"c619c67a-04e0-45a2-8759-4f278b5b3326\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:35.961513 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.961447    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" event={"ID":"91aab7bc-5ef8-4037-a001-3fefd605f6b4","Type":"ContainerStarted","Data":"93a9c0ed4ac74e1f0bcc1dce77e1b322efa4cd805c9b9c7008a192eaacf59ae8"}
Nov 02 13:54:35.961513 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.961501    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" event={"ID":"91aab7bc-5ef8-4037-a001-3fefd605f6b4","Type":"ContainerStarted","Data":"afa0774e1077e07a32afded7fb2f762fedb37372fae829f2c7f24511e41112f7"}
Nov 02 13:54:35.995581 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:35.995555    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:36.017188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.015063    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" podStartSLOduration=2.015043956 podStartE2EDuration="2.015043956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:36.013275771 +0000 UTC m=+666.764580208" watchObservedRunningTime="2026-11-02 13:54:36.015043956 +0000 UTC m=+666.766348389"
Nov 02 13:54:36.312011 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:36.311980    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc619c67a_04e0_45a2_8759_4f278b5b3326.slice/crio-6ae7d5f9e14345f85b98cefda5a8fe42169f1086dc6d3178e962318937bd8549 WatchSource:0}: Error finding container 6ae7d5f9e14345f85b98cefda5a8fe42169f1086dc6d3178e962318937bd8549: Status 404 returned error can't find the container with id 6ae7d5f9e14345f85b98cefda5a8fe42169f1086dc6d3178e962318937bd8549
Nov 02 13:54:36.312773 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.312749    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"]
Nov 02 13:54:36.575227 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.575152    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"]
Nov 02 13:54:36.578875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.578859    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:36.599805 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.599700    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-ad42e0\""
Nov 02 13:54:36.599941 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.599808    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-ad42e0-dockercfg-j4f6h\""
Nov 02 13:54:36.607387 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.607363    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"]
Nov 02 13:54:36.631857 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.631757    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nh7j\" (UniqueName: \"kubernetes.io/projected/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-kube-api-access-5nh7j\") pod \"payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q\" (UID: \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:36.632174 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.631979    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q\" (UID: \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:36.705604 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.705564    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"]
Nov 02 13:54:36.713247 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.713217    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:36.714386 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.714354    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"]
Nov 02 13:54:36.732003 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.731976    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:36.733465 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.733437    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nh7j\" (UniqueName: \"kubernetes.io/projected/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-kube-api-access-5nh7j\") pod \"payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q\" (UID: \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:36.733581 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.733566    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q\" (UID: \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:36.734233 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.734210    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q\" (UID: \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:36.737089 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.737066    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:36.757579 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.757557    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nh7j\" (UniqueName: \"kubernetes.io/projected/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-kube-api-access-5nh7j\") pod \"payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q\" (UID: \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:36.834730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.834650    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4498b\" (UniqueName: \"kubernetes.io/projected/2358bb25-5f5b-42a4-9678-852be7940e34-kube-api-access-4498b\") pod \"payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw\" (UID: \"2358bb25-5f5b-42a4-9678-852be7940e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:36.834882 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.834828    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2358bb25-5f5b-42a4-9678-852be7940e34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw\" (UID: \"2358bb25-5f5b-42a4-9678-852be7940e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:36.891623 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.891595    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:36.935902 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.935864    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2358bb25-5f5b-42a4-9678-852be7940e34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw\" (UID: \"2358bb25-5f5b-42a4-9678-852be7940e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:36.936034 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.935990    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4498b\" (UniqueName: \"kubernetes.io/projected/2358bb25-5f5b-42a4-9678-852be7940e34-kube-api-access-4498b\") pod \"payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw\" (UID: \"2358bb25-5f5b-42a4-9678-852be7940e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:36.936598 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.936574    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2358bb25-5f5b-42a4-9678-852be7940e34-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw\" (UID: \"2358bb25-5f5b-42a4-9678-852be7940e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:36.960348 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.960323    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4498b\" (UniqueName: \"kubernetes.io/projected/2358bb25-5f5b-42a4-9678-852be7940e34-kube-api-access-4498b\") pod \"payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw\" (UID: \"2358bb25-5f5b-42a4-9678-852be7940e34\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:36.969373 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.969340    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z" event={"ID":"c619c67a-04e0-45a2-8759-4f278b5b3326","Type":"ContainerStarted","Data":"109e2d75d713964eb4bb00d30d4ac07cf9e4608c9a27a3798e2ecfacdc5dcf05"}
Nov 02 13:54:36.969492 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.969409    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:36.969492 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.969432    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z" event={"ID":"c619c67a-04e0-45a2-8759-4f278b5b3326","Type":"ContainerStarted","Data":"6ae7d5f9e14345f85b98cefda5a8fe42169f1086dc6d3178e962318937bd8549"}
Nov 02 13:54:36.969671 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.969654    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:36.970683 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:36.970661    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:54:37.013456 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:37.013412    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z" podStartSLOduration=2.013395932 podStartE2EDuration="2.013395932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:37.01194138 +0000 UTC m=+667.763245818" watchObservedRunningTime="2026-11-02 13:54:37.013395932 +0000 UTC m=+667.764700369"
Nov 02 13:54:37.029349 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:37.029325    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:37.191060 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:37.191003    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28fcc5d_ad6f_426e_a370_5f6576bf37d6.slice/crio-10b71409d184378746825d4c7cc6c7917f32c09d391fa28e222c8c9aa48642e5 WatchSource:0}: Error finding container 10b71409d184378746825d4c7cc6c7917f32c09d391fa28e222c8c9aa48642e5: Status 404 returned error can't find the container with id 10b71409d184378746825d4c7cc6c7917f32c09d391fa28e222c8c9aa48642e5
Nov 02 13:54:37.192408 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:37.192378    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"]
Nov 02 13:54:37.371027 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:37.370839    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"]
Nov 02 13:54:37.978069 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:37.978037    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q" event={"ID":"e28fcc5d-ad6f-426e-a370-5f6576bf37d6","Type":"ContainerStarted","Data":"117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab"}
Nov 02 13:54:37.978261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:37.978104    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:37.978261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:37.978129    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q" event={"ID":"e28fcc5d-ad6f-426e-a370-5f6576bf37d6","Type":"ContainerStarted","Data":"10b71409d184378746825d4c7cc6c7917f32c09d391fa28e222c8c9aa48642e5"}
Nov 02 13:54:37.979747 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:37.979717    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw" event={"ID":"2358bb25-5f5b-42a4-9678-852be7940e34","Type":"ContainerStarted","Data":"bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685"}
Nov 02 13:54:37.979898 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:37.979759    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw" event={"ID":"2358bb25-5f5b-42a4-9678-852be7940e34","Type":"ContainerStarted","Data":"f11e2b36adba1219225e1bdaeeb6b54a019d7d2c357642c2ce1158713641a8c4"}
Nov 02 13:54:38.012732 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.012700    5034 patch_prober.go:29] interesting pod/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:54:38.012873 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.012755    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:54:38.022693 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.022649    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q" podStartSLOduration=2.022635964 podStartE2EDuration="2.022635964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:38.021060253 +0000 UTC m=+668.772364691" watchObservedRunningTime="2026-11-02 13:54:38.022635964 +0000 UTC m=+668.773940458"
Nov 02 13:54:38.069406 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.069365    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw" podStartSLOduration=2.069353638 podStartE2EDuration="2.069353638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:38.067227581 +0000 UTC m=+668.818532020" watchObservedRunningTime="2026-11-02 13:54:38.069353638 +0000 UTC m=+668.820658108"
Nov 02 13:54:38.426266 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.426237    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:54:38.551942 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.551869    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-credential-socket\") pod \"0678ee83-5d09-4d17-919a-942ba88e5d16\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") "
Nov 02 13:54:38.551942 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.551912    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-data\") pod \"0678ee83-5d09-4d17-919a-942ba88e5d16\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") "
Nov 02 13:54:38.552138 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.551975    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-envoy\") pod \"0678ee83-5d09-4d17-919a-942ba88e5d16\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") "
Nov 02 13:54:38.552138 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552015    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0678ee83-5d09-4d17-919a-942ba88e5d16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0678ee83-5d09-4d17-919a-942ba88e5d16-istiod-ca-cert\") pod \"0678ee83-5d09-4d17-919a-942ba88e5d16\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") "
Nov 02 13:54:38.552138 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552054    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-certs\") pod \"0678ee83-5d09-4d17-919a-942ba88e5d16\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") "
Nov 02 13:54:38.552138 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552072    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-token\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-token\") pod \"0678ee83-5d09-4d17-919a-942ba88e5d16\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") "
Nov 02 13:54:38.552138 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552092    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-kube-api-access-gnbgb\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-kube-api-access-gnbgb\") pod \"0678ee83-5d09-4d17-919a-942ba88e5d16\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") "
Nov 02 13:54:38.552392 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552151    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-credential-socket" pod "0678ee83-5d09-4d17-919a-942ba88e5d16" (UID: "0678ee83-5d09-4d17-919a-942ba88e5d16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:38.552392 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552159    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-socket\") pod \"0678ee83-5d09-4d17-919a-942ba88e5d16\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") "
Nov 02 13:54:38.552392 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552253    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-podinfo\") pod \"0678ee83-5d09-4d17-919a-942ba88e5d16\" (UID: \"0678ee83-5d09-4d17-919a-942ba88e5d16\") "
Nov 02 13:54:38.552392 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552278    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-data" pod "0678ee83-5d09-4d17-919a-942ba88e5d16" (UID: "0678ee83-5d09-4d17-919a-942ba88e5d16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:38.552605 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552403    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-certs" pod "0678ee83-5d09-4d17-919a-942ba88e5d16" (UID: "0678ee83-5d09-4d17-919a-942ba88e5d16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:38.552605 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552434    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0678ee83-5d09-4d17-919a-942ba88e5d16-istiod-ca-cert" pod "0678ee83-5d09-4d17-919a-942ba88e5d16" (UID: "0678ee83-5d09-4d17-919a-942ba88e5d16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:54:38.552605 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552519    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-socket" pod "0678ee83-5d09-4d17-919a-942ba88e5d16" (UID: "0678ee83-5d09-4d17-919a-942ba88e5d16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:38.552773 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552750    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:38.552872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552806    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:38.552872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552823    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:38.552872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552835    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0678ee83-5d09-4d17-919a-942ba88e5d16-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:38.552872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.552848    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:38.554343 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.554314    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-envoy" pod "0678ee83-5d09-4d17-919a-942ba88e5d16" (UID: "0678ee83-5d09-4d17-919a-942ba88e5d16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:54:38.554453 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.554429    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-podinfo" pod "0678ee83-5d09-4d17-919a-942ba88e5d16" (UID: "0678ee83-5d09-4d17-919a-942ba88e5d16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:54:38.554453 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.554440    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-token" pod "0678ee83-5d09-4d17-919a-942ba88e5d16" (UID: "0678ee83-5d09-4d17-919a-942ba88e5d16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:38.554548 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.554464    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-kube-api-access-gnbgb" pod "0678ee83-5d09-4d17-919a-942ba88e5d16" (UID: "0678ee83-5d09-4d17-919a-942ba88e5d16"). InnerVolumeSpecName "kube-api-access-gnbgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:54:38.654320 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.654295    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:38.654320 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.654317    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:38.654320 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.654326    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:38.654519 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.654334    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnbgb\" (UniqueName: \"kubernetes.io/projected/0678ee83-5d09-4d17-919a-942ba88e5d16-kube-api-access-gnbgb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:54:38.988690 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.988654    5034 generic.go:356] "Generic (PLEG): container finished" podID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerID="dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1" exitCode=0
Nov 02 13:54:38.988894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.988704    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"
Nov 02 13:54:38.988894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.988760    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" event={"ID":"0678ee83-5d09-4d17-919a-942ba88e5d16","Type":"ContainerDied","Data":"dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1"}
Nov 02 13:54:38.988894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.988876    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2" event={"ID":"0678ee83-5d09-4d17-919a-942ba88e5d16","Type":"ContainerDied","Data":"930604dcd974abe953ce0fe8630b780ca5b0147434c32a870546dbf0f71ab47f"}
Nov 02 13:54:38.989072 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.988915    5034 scope.go:122] "RemoveContainer" containerID="dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1"
Nov 02 13:54:38.989681 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.989658    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:38.999890 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:38.999873    5034 scope.go:122] "RemoveContainer" containerID="9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5"
Nov 02 13:54:39.009044 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:39.009025    5034 scope.go:122] "RemoveContainer" containerID="dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1"
Nov 02 13:54:39.009324 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:39.009301    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1\": container with ID starting with dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1 not found: ID does not exist" containerID="dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1"
Nov 02 13:54:39.009443 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:39.009332    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1"} err="failed to get container status \"dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1\": rpc error: code = NotFound desc = could not find container \"dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1\": container with ID starting with dae6e59bd495ae6587d8e83530a3883e9603a5404b57dd157d22d047440fe9b1 not found: ID does not exist"
Nov 02 13:54:39.009443 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:39.009357    5034 scope.go:122] "RemoveContainer" containerID="9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5"
Nov 02 13:54:39.009661 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:39.009636    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5\": container with ID starting with 9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5 not found: ID does not exist" containerID="9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5"
Nov 02 13:54:39.009769 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:39.009665    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5"} err="failed to get container status \"9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5\": rpc error: code = NotFound desc = could not find container \"9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5\": container with ID starting with 9f985b3545130ea97c31b6b95c4f9d591f08cb17958708fa44b8eca0e731fac5 not found: ID does not exist"
Nov 02 13:54:39.076801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:39.076022    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"]
Nov 02 13:54:39.084073 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:39.084042    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e7818088-openshift-default-64587b6d9-7p2b2"]
Nov 02 13:54:39.395684 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:39.395618    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" path="/var/lib/kubelet/pods/0678ee83-5d09-4d17-919a-942ba88e5d16/volumes"
Nov 02 13:54:40.348886 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.348766    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"]
Nov 02 13:54:40.349382 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.349354    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy"
Nov 02 13:54:40.349382 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.349374    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy"
Nov 02 13:54:40.349508 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.349395    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy"
Nov 02 13:54:40.349508 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.349401    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy"
Nov 02 13:54:40.349508 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.349483    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy"
Nov 02 13:54:40.353689 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.353667    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:40.429802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.429074    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-92f249-dockercfg-ff8pn\""
Nov 02 13:54:40.429802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.429396    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-92f249\""
Nov 02 13:54:40.447837 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.436274    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"]
Nov 02 13:54:40.447837 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.440870    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb7fw\" (UniqueName: \"kubernetes.io/projected/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-kube-api-access-sb7fw\") pod \"payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj\" (UID: \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:40.447837 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.441053    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj\" (UID: \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:40.508654 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.508619    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"]
Nov 02 13:54:40.509788 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.509745    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0678ee83-5d09-4d17-919a-942ba88e5d16" containerName="istio-proxy"
Nov 02 13:54:40.513965 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.513893    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:40.516344 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.516134    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"]
Nov 02 13:54:40.541940 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.541913    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj\" (UID: \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:40.542047 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.541965    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb7fw\" (UniqueName: \"kubernetes.io/projected/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-kube-api-access-sb7fw\") pod \"payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj\" (UID: \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:40.542634 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.542616    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj\" (UID: \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:40.568475 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.568451    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb7fw\" (UniqueName: \"kubernetes.io/projected/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-kube-api-access-sb7fw\") pod \"payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj\" (UID: \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:40.643481 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.643403    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnm9\" (UniqueName: \"kubernetes.io/projected/39520cdc-cdca-4c22-81ef-480183b2d652-kube-api-access-jjnm9\") pod \"payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv\" (UID: \"39520cdc-cdca-4c22-81ef-480183b2d652\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:40.643634 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.643490    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39520cdc-cdca-4c22-81ef-480183b2d652-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv\" (UID: \"39520cdc-cdca-4c22-81ef-480183b2d652\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:40.667961 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.667938    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:40.744436 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.744397    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39520cdc-cdca-4c22-81ef-480183b2d652-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv\" (UID: \"39520cdc-cdca-4c22-81ef-480183b2d652\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:40.746222 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.745939    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnm9\" (UniqueName: \"kubernetes.io/projected/39520cdc-cdca-4c22-81ef-480183b2d652-kube-api-access-jjnm9\") pod \"payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv\" (UID: \"39520cdc-cdca-4c22-81ef-480183b2d652\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:40.746526 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.746499    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39520cdc-cdca-4c22-81ef-480183b2d652-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv\" (UID: \"39520cdc-cdca-4c22-81ef-480183b2d652\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:40.802802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.794484    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnm9\" (UniqueName: \"kubernetes.io/projected/39520cdc-cdca-4c22-81ef-480183b2d652-kube-api-access-jjnm9\") pod \"payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv\" (UID: \"39520cdc-cdca-4c22-81ef-480183b2d652\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:40.837132 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:40.837095    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:41.004201 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:41.004168    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"]
Nov 02 13:54:41.004567 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:41.004543    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e58a1f_e4b4_44f6_9b75_0fb12c933ff8.slice/crio-2fecb54a81f3057afbbafd75dc964af6e27a298e916ff49ee809d46032dcae6b WatchSource:0}: Error finding container 2fecb54a81f3057afbbafd75dc964af6e27a298e916ff49ee809d46032dcae6b: Status 404 returned error can't find the container with id 2fecb54a81f3057afbbafd75dc964af6e27a298e916ff49ee809d46032dcae6b
Nov 02 13:54:41.172126 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:41.172097    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"]
Nov 02 13:54:42.023640 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:42.023601    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv" event={"ID":"39520cdc-cdca-4c22-81ef-480183b2d652","Type":"ContainerStarted","Data":"58c9471215efcafc5c11fa233163667d1cc42e84a0d68993aa12dd3311f624e8"}
Nov 02 13:54:42.025152 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:42.025125    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv" event={"ID":"39520cdc-cdca-4c22-81ef-480183b2d652","Type":"ContainerStarted","Data":"2ace25600c64b2fd76b2f246037d959362a8310c63bc82408eb84104fed2e64f"}
Nov 02 13:54:42.025331 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:42.025319    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:42.027504 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:42.027484    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj" event={"ID":"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8","Type":"ContainerStarted","Data":"a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380"}
Nov 02 13:54:42.027652 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:42.027638    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj" event={"ID":"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8","Type":"ContainerStarted","Data":"2fecb54a81f3057afbbafd75dc964af6e27a298e916ff49ee809d46032dcae6b"}
Nov 02 13:54:42.028056 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:42.028020    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:42.068263 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:42.068226    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv" podStartSLOduration=2.068214201 podStartE2EDuration="2.068214201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:42.065916819 +0000 UTC m=+672.817221249" watchObservedRunningTime="2026-11-02 13:54:42.068214201 +0000 UTC m=+672.819518685"
Nov 02 13:54:42.126046 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:42.125977    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj" podStartSLOduration=2.125956855 podStartE2EDuration="2.125956855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:42.112053008 +0000 UTC m=+672.863357447" watchObservedRunningTime="2026-11-02 13:54:42.125956855 +0000 UTC m=+672.877261293"
Nov 02 13:54:44.957684 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:44.957658    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 13:54:44.958115 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:44.957726    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 13:54:47.983294 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:47.983263    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 13:54:48.418686 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:54:48.418594    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:54:48.992578 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:48.992540    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 13:54:50.005403 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:50.005374    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 13:54:50.097035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:50.097001    5034 generic.go:356] "Generic (PLEG): container finished" podID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerID="357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5" exitCode=0
Nov 02 13:54:50.097219 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:50.097069    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" event={"ID":"856cd9ab-9a6f-4ba3-b8ea-87b83d868832","Type":"ContainerDied","Data":"357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5"}
Nov 02 13:54:50.097478 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:50.097459    5034 scope.go:122] "RemoveContainer" containerID="357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5"
Nov 02 13:54:50.098990 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:50.098965    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:50.099124 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:50.099027    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:50.099124 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:50.099070    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:51.111351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.111314    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" event={"ID":"856cd9ab-9a6f-4ba3-b8ea-87b83d868832","Type":"ContainerStarted","Data":"fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3"}
Nov 02 13:54:51.113591 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.113564    5034 generic.go:356] "Generic (PLEG): container finished" podID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerID="f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df" exitCode=0
Nov 02 13:54:51.113741 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.113631    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" event={"ID":"8b9df9c2-2c16-4bb5-bb05-ee1667367dba","Type":"ContainerDied","Data":"f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df"}
Nov 02 13:54:51.113973 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.113958    5034 scope.go:122] "RemoveContainer" containerID="f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df"
Nov 02 13:54:51.115212 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.115185    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:51.115284 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.115248    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:51.115284 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.115278    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:54:51.488217 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.488182    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:51.488542 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.488518    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 13:54:51.492832 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:51.488795    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 13:54:52.147944 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:52.147897    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" event={"ID":"8b9df9c2-2c16-4bb5-bb05-ee1667367dba","Type":"ContainerStarted","Data":"d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff"}
Nov 02 13:54:52.447171 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:52.447072    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:52.448565 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:52.448537    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 02 13:54:52.448725 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:52.448594    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 02 13:54:52.487367 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:52.487334    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 13:54:52.487660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:52.487630    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 13:54:53.057806 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:53.057493    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 13:54:53.058654 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:53.058630    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 13:54:53.540863 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:53.474536    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 02 13:54:53.592062 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:53.511171    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 13:54:53.592062 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:53.591629    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 13:54:53.592421 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:53.592374    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 02 13:54:54.447518 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:54.447476    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 02 13:54:54.447920 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:54.447889    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 02 13:54:54.495215 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:54.495182    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 13:54:54.495501 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:54.495461    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 13:54:55.451203 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.451165    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 02 13:54:55.452168 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.452116    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 02 13:54:55.490213 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.490179    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Nov 02 13:54:55.490494 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.490450    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Nov 02 13:54:55.950429 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.950394    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:55.955841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.955812    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"]
Nov 02 13:54:55.969810 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.967594    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"]
Nov 02 13:54:55.969810 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.967699    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.005813 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.989860    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"openshift-service-ca.crt\""
Nov 02 13:54:56.011104 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:55.995979    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"kube-root-ca.crt\""
Nov 02 13:54:56.043841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.043551    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"default-dockercfg-9z88n\""
Nov 02 13:54:56.043841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.043637    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-0969b1\"/\"simulator-w2-0969b1-kserve-self-signed-certs\""
Nov 02 13:54:56.231871 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.204359    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-dshm\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.232205 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.232174    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tmp-dir\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.232363 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.232346    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-home\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.232620 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.232603    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tls-certs\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.242828 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.232756    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcvtb\" (UniqueName: \"kubernetes.io/projected/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-kube-api-access-zcvtb\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.251491 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.251460    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-model-cache\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.354974 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.354935    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tls-certs\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.355185 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.354985    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcvtb\" (UniqueName: \"kubernetes.io/projected/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-kube-api-access-zcvtb\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.355185 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.355015    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-model-cache\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.355185 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.355107    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-dshm\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.355185 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.355134    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tmp-dir\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.355185 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.355150    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-home\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.355530 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.355513    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-home\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.356059 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.356029    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-model-cache\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.356245 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.356177    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tmp-dir\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.358495 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.358470    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-dshm\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.358666 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.358569    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tls-certs\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.390695 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.390653    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcvtb\" (UniqueName: \"kubernetes.io/projected/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-kube-api-access-zcvtb\") pod \"simulator-w2-0969b1-kserve-698b79d64d-s5chb\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") " pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.447855 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.447821    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 02 13:54:56.448127 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.448097    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 02 13:54:56.558118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.558036    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:56.566695 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.566669    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:56.567966 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.567947    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:56.573064 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.573037    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 13:54:56.932626 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:56.932533    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"]
Nov 02 13:54:56.948473 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:54:56.948446    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d1d8c55_9c6e_435b_8f68_36b3fbedb914.slice/crio-1b1ea9831a630a1cd4618d8d58120c93e85bb952049adad46c7b7eaa12144222 WatchSource:0}: Error finding container 1b1ea9831a630a1cd4618d8d58120c93e85bb952049adad46c7b7eaa12144222: Status 404 returned error can't find the container with id 1b1ea9831a630a1cd4618d8d58120c93e85bb952049adad46c7b7eaa12144222
Nov 02 13:54:57.354976 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:57.354937    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" event={"ID":"8d1d8c55-9c6e-435b-8f68-36b3fbedb914","Type":"ContainerStarted","Data":"9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c"}
Nov 02 13:54:57.355198 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:57.354991    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" event={"ID":"8d1d8c55-9c6e-435b-8f68-36b3fbedb914","Type":"ContainerStarted","Data":"1b1ea9831a630a1cd4618d8d58120c93e85bb952049adad46c7b7eaa12144222"}
Nov 02 13:54:57.448015 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:57.447980    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Nov 02 13:54:57.448176 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:57.448034    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Nov 02 13:54:57.558931 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:57.558893    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:54:58.451103 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:58.451077    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:58.451981 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:58.451964    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 13:54:58.537101 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:54:58.537045    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podStartSLOduration=3.5370320729999998 podStartE2EDuration="3.537032073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:54:57.408055409 +0000 UTC m=+688.159359849" watchObservedRunningTime="2026-11-02 13:54:58.537032073 +0000 UTC m=+689.288336511"
Nov 02 13:55:02.432986 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:02.432951    5034 generic.go:356] "Generic (PLEG): container finished" podID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerID="93a9c0ed4ac74e1f0bcc1dce77e1b322efa4cd805c9b9c7008a192eaacf59ae8" exitCode=0
Nov 02 13:55:02.433441 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:02.432999    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" event={"ID":"91aab7bc-5ef8-4037-a001-3fefd605f6b4","Type":"ContainerDied","Data":"93a9c0ed4ac74e1f0bcc1dce77e1b322efa4cd805c9b9c7008a192eaacf59ae8"}
Nov 02 13:55:02.433441 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:02.433310    5034 scope.go:122] "RemoveContainer" containerID="93a9c0ed4ac74e1f0bcc1dce77e1b322efa4cd805c9b9c7008a192eaacf59ae8"
Nov 02 13:55:02.434720 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:02.434685    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:55:02.434844 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:02.434826    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:55:02.434895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:02.434880    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:55:03.440811 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:03.440765    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" event={"ID":"91aab7bc-5ef8-4037-a001-3fefd605f6b4","Type":"ContainerStarted","Data":"5eb0d731dd9ca14436163bf8c66641565a137566ccdc585e4591f1d8ab9eef7f"}
Nov 02 13:55:03.731344 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:03.731319    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:55:03.732519 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:03.732498    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 13:55:03.732574 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:03.732545    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 13:55:04.732621 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:04.732587    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 13:55:04.733027 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:04.732665    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 13:55:05.732393 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:05.732360    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 13:55:05.732563 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:05.732437    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 13:55:06.558680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:06.558644    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:55:06.599292 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:06.599244    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:55:06.732552 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:06.732521    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 13:55:06.732703 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:06.732569    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 13:55:06.977951 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:06.977861    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:55:07.511271 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:07.511235    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:55:07.732546 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:07.732509    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Nov 02 13:55:07.733053 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:07.732565    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Nov 02 13:55:08.735743 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:08.735712    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:55:08.736895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:08.736874    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 13:55:10.855543 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.855511    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"]
Nov 02 13:55:10.859640 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.859615    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:10.889844 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.889818    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ad42e0\"/\"kube-root-ca.crt\""
Nov 02 13:55:10.890111 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.890087    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-ad42e0\"/\"openshift-service-ca.crt\""
Nov 02 13:55:10.890228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.890093    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad42e0\"/\"default-dockercfg-vcw6v\""
Nov 02 13:55:10.890228 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.890122    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad42e0\"/\"simulator-w4-ad42e0-kserve-self-signed-certs\""
Nov 02 13:55:10.892855 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.892834    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"]
Nov 02 13:55:10.914897 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.914874    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b39766e-2b9e-4941-bf05-9ea55c506925-tls-certs\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:10.915003 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.914907    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-tmp-dir\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:10.915078 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.915034    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-model-cache\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:10.915138 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.915115    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcpv8\" (UniqueName: \"kubernetes.io/projected/5b39766e-2b9e-4941-bf05-9ea55c506925-kube-api-access-dcpv8\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:10.915196 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.915144    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-dshm\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:10.915292 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:10.915273    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-home\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.016441 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.016410    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b39766e-2b9e-4941-bf05-9ea55c506925-tls-certs\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.016441 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.016444    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-tmp-dir\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.016643 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.016485    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-model-cache\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.016643 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.016518    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcpv8\" (UniqueName: \"kubernetes.io/projected/5b39766e-2b9e-4941-bf05-9ea55c506925-kube-api-access-dcpv8\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.016643 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.016534    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-dshm\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.016643 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.016576    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-home\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.016921 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.016900    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-tmp-dir\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.016991 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.016967    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-model-cache\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.017067 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.017049    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-home\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.018774 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.018756    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-dshm\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.019122 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.019107    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b39766e-2b9e-4941-bf05-9ea55c506925-tls-certs\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.040917 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.040891    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcpv8\" (UniqueName: \"kubernetes.io/projected/5b39766e-2b9e-4941-bf05-9ea55c506925-kube-api-access-dcpv8\") pod \"simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") " pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.173172 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.173068    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:11.473389 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.473360    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"]
Nov 02 13:55:11.473515 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:55:11.473376    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b39766e_2b9e_4941_bf05_9ea55c506925.slice/crio-125552a4ef5bde955b72db76bbeeb47b0fde4c403612ee8a30ac3596c0564815 WatchSource:0}: Error finding container 125552a4ef5bde955b72db76bbeeb47b0fde4c403612ee8a30ac3596c0564815: Status 404 returned error can't find the container with id 125552a4ef5bde955b72db76bbeeb47b0fde4c403612ee8a30ac3596c0564815
Nov 02 13:55:11.512802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:11.512754    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" event={"ID":"5b39766e-2b9e-4941-bf05-9ea55c506925","Type":"ContainerStarted","Data":"125552a4ef5bde955b72db76bbeeb47b0fde4c403612ee8a30ac3596c0564815"}
Nov 02 13:55:12.529101 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:12.529068    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" event={"ID":"5b39766e-2b9e-4941-bf05-9ea55c506925","Type":"ContainerStarted","Data":"2eedf28857326edb415485db95aa14595d94103f72390c4574d2eeaf03cc6ec4"}
Nov 02 13:55:12.572624 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:12.572575    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podStartSLOduration=2.572563439 podStartE2EDuration="2.572563439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:12.571386621 +0000 UTC m=+703.322691062" watchObservedRunningTime="2026-11-02 13:55:12.572563439 +0000 UTC m=+703.323867871"
Nov 02 13:55:13.173852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:13.173815    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:18.415751 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:55:18.415708    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:55:21.174006 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:21.173968    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:21.186523 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:21.186495    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:21.595476 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:21.595438    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:55:22.582610 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.582570    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"]
Nov 02 13:55:22.593768 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.593742    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"]
Nov 02 13:55:22.594078 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.594064    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.610191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.610163    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"default-dockercfg-clf8h\""
Nov 02 13:55:22.610330 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.610286    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"openshift-service-ca.crt\""
Nov 02 13:55:22.622517 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.622495    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"premium-w0-1a562c-kserve-self-signed-certs\""
Nov 02 13:55:22.622679 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.622533    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1a562c\"/\"kube-root-ca.crt\""
Nov 02 13:55:22.753716 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.753682    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-dshm\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.753933 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.753792    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-home\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.753933 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.753812    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-model-cache\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.753933 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.753868    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqx7d\" (UniqueName: \"kubernetes.io/projected/6949af01-2c4c-4d4d-9bd5-02724101b868-kube-api-access-mqx7d\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.754096 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.754022    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6949af01-2c4c-4d4d-9bd5-02724101b868-tls-certs\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.754096 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.754076    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-tmp-dir\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.854733 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.854655    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-home\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.854733 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.854687    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-model-cache\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.854733 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.854708    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqx7d\" (UniqueName: \"kubernetes.io/projected/6949af01-2c4c-4d4d-9bd5-02724101b868-kube-api-access-mqx7d\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.855031 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.854807    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6949af01-2c4c-4d4d-9bd5-02724101b868-tls-certs\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.855031 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.854854    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-tmp-dir\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.855031 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.854910    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-dshm\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.855199 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.855140    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-home\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.855254 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.855194    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-model-cache\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.855306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.855262    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-tmp-dir\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.857057 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.857036    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-dshm\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.857410 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.857393    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6949af01-2c4c-4d4d-9bd5-02724101b868-tls-certs\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.882176 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.882146    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqx7d\" (UniqueName: \"kubernetes.io/projected/6949af01-2c4c-4d4d-9bd5-02724101b868-kube-api-access-mqx7d\") pod \"premium-w0-1a562c-kserve-5c7c64476-5pdrm\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") " pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:22.911008 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:22.910981    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:23.261163 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:55:23.261131    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6949af01_2c4c_4d4d_9bd5_02724101b868.slice/crio-0412fa698b03801a883f52cd52894762eec3c08d4177a2216f56e6973f3358fe WatchSource:0}: Error finding container 0412fa698b03801a883f52cd52894762eec3c08d4177a2216f56e6973f3358fe: Status 404 returned error can't find the container with id 0412fa698b03801a883f52cd52894762eec3c08d4177a2216f56e6973f3358fe
Nov 02 13:55:23.261767 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:23.261743    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"]
Nov 02 13:55:23.611613 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:23.611531    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" event={"ID":"6949af01-2c4c-4d4d-9bd5-02724101b868","Type":"ContainerStarted","Data":"13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17"}
Nov 02 13:55:23.611613 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:23.611576    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" event={"ID":"6949af01-2c4c-4d4d-9bd5-02724101b868","Type":"ContainerStarted","Data":"0412fa698b03801a883f52cd52894762eec3c08d4177a2216f56e6973f3358fe"}
Nov 02 13:55:23.655116 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:23.655066    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podStartSLOduration=1.655054372 podStartE2EDuration="1.655054372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:23.654383696 +0000 UTC m=+714.405688136" watchObservedRunningTime="2026-11-02 13:55:23.655054372 +0000 UTC m=+714.406358806"
Nov 02 13:55:23.911368 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:23.911297    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:32.911832 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:32.911792    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:32.924610 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:32.924586    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:33.692925 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:33.692896    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:55:39.278980 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.278943    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"]
Nov 02 13:55:39.285186 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.285154    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.302420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.302395    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"]
Nov 02 13:55:39.312282 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.312261    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"default-dockercfg-b9x6d\""
Nov 02 13:55:39.312400 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.312383    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"openshift-service-ca.crt\""
Nov 02 13:55:39.312691 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.312675    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"premium-w1-92f249-kserve-self-signed-certs\""
Nov 02 13:55:39.314827 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.314808    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"kube-root-ca.crt\""
Nov 02 13:55:39.345391 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.345369    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-dshm\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.345506 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.345408    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tmp-dir\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.345506 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.345490    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22xsb\" (UniqueName: \"kubernetes.io/projected/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-kube-api-access-22xsb\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.345592 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.345530    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tls-certs\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.345592 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.345553    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-home\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.345592 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.345572    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-model-cache\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.446211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.446183    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22xsb\" (UniqueName: \"kubernetes.io/projected/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-kube-api-access-22xsb\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.446348 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.446221    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tls-certs\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.446348 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.446239    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-home\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.446348 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.446257    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-model-cache\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.446489 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.446409    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-dshm\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.446489 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.446434    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tmp-dir\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.446651 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.446629    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-home\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.446742 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.446703    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-model-cache\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.446742 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.446725    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tmp-dir\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.448523 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.448501    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-dshm\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.448619 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.448602    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tls-certs\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.484460 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.484422    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22xsb\" (UniqueName: \"kubernetes.io/projected/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-kube-api-access-22xsb\") pod \"premium-w1-92f249-kserve-767f74bf5c-lvl6s\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") " pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.599351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.599266    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:39.899537 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:55:39.899501    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod903cf918_c71c_42c4_8fb4_12ebd76d9b6f.slice/crio-dacf527616de266e230d41ad413e8aaf46e7e9299d3ec35f77f30c974b9d9a28 WatchSource:0}: Error finding container dacf527616de266e230d41ad413e8aaf46e7e9299d3ec35f77f30c974b9d9a28: Status 404 returned error can't find the container with id dacf527616de266e230d41ad413e8aaf46e7e9299d3ec35f77f30c974b9d9a28
Nov 02 13:55:39.900191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:39.900166    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"]
Nov 02 13:55:40.730849 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:40.730812    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" event={"ID":"903cf918-c71c-42c4-8fb4-12ebd76d9b6f","Type":"ContainerStarted","Data":"c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb"}
Nov 02 13:55:40.731257 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:40.730855    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" event={"ID":"903cf918-c71c-42c4-8fb4-12ebd76d9b6f","Type":"ContainerStarted","Data":"dacf527616de266e230d41ad413e8aaf46e7e9299d3ec35f77f30c974b9d9a28"}
Nov 02 13:55:40.774000 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:40.773951    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podStartSLOduration=1.773936576 podStartE2EDuration="1.773936576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:40.772258335 +0000 UTC m=+731.523562773" watchObservedRunningTime="2026-11-02 13:55:40.773936576 +0000 UTC m=+731.525241013"
Nov 02 13:55:41.599370 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:41.599343    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:48.413548 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:55:48.413506    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:55:49.599639 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.599593    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:49.612577 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.612551    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:49.796363 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.796332    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:55:49.838944 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.838911    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"]
Nov 02 13:55:49.842988 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.842920    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:49.864793 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.864754    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"]
Nov 02 13:55:49.869997 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.869943    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad42e0\"/\"premium-w4-ad42e0-kserve-self-signed-certs\""
Nov 02 13:55:49.974641 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.974607    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-dshm\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:49.974641 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.974645    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-home\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:49.974927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.974772    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-model-cache\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:49.974927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.974821    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xnmx\" (UniqueName: \"kubernetes.io/projected/b2c0415b-c21f-4867-80cc-5bb953cb0532-kube-api-access-2xnmx\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:49.975036 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.974944    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-tmp-dir\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:49.975036 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:49.974971    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c0415b-c21f-4867-80cc-5bb953cb0532-tls-certs\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.076435 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.076395    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-tmp-dir\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.076648 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.076442    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c0415b-c21f-4867-80cc-5bb953cb0532-tls-certs\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.076648 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.076573    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-dshm\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.076648 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.076605    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-home\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.076888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.076703    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-model-cache\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.076888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.076730    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xnmx\" (UniqueName: \"kubernetes.io/projected/b2c0415b-c21f-4867-80cc-5bb953cb0532-kube-api-access-2xnmx\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.076888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.076869    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-tmp-dir\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.077140 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.077120    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-home\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.077232 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.077210    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-model-cache\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.080054 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.080017    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c0415b-c21f-4867-80cc-5bb953cb0532-tls-certs\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.082049 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.082009    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-dshm\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.100164 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.100139    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xnmx\" (UniqueName: \"kubernetes.io/projected/b2c0415b-c21f-4867-80cc-5bb953cb0532-kube-api-access-2xnmx\") pod \"premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") " pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.166243 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.166187    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:55:50.481202 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.481169    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"]
Nov 02 13:55:50.481466 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:55:50.481442    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2c0415b_c21f_4867_80cc_5bb953cb0532.slice/crio-ca382985524d9230d515869b21d112fd6849e0b45b6829914cb56c7325269461 WatchSource:0}: Error finding container ca382985524d9230d515869b21d112fd6849e0b45b6829914cb56c7325269461: Status 404 returned error can't find the container with id ca382985524d9230d515869b21d112fd6849e0b45b6829914cb56c7325269461
Nov 02 13:55:50.793200 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.793165    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" event={"ID":"b2c0415b-c21f-4867-80cc-5bb953cb0532","Type":"ContainerStarted","Data":"8c50f3514fc4dae2351193feaa07abfc5b23bf0df630d425b99390d59258c47d"}
Nov 02 13:55:50.793690 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.793217    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" event={"ID":"b2c0415b-c21f-4867-80cc-5bb953cb0532","Type":"ContainerStarted","Data":"ca382985524d9230d515869b21d112fd6849e0b45b6829914cb56c7325269461"}
Nov 02 13:55:50.840145 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:50.840094    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podStartSLOduration=1.8400782329999998 podStartE2EDuration="1.840078233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:55:50.838032858 +0000 UTC m=+741.589337287" watchObservedRunningTime="2026-11-02 13:55:50.840078233 +0000 UTC m=+741.591382670"
Nov 02 13:55:51.167009 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:55:51.166928    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:56:00.166464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:00.166433    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:56:00.179679 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:00.179652    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:56:00.869485 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:00.869453    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 13:56:10.465479 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.465400    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"]
Nov 02 13:56:10.470697 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.470675    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.493124 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.493098    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"]
Nov 02 13:56:10.500980 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.500957    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:10.501190 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.501103    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:10.501190 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.501181    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:56:10.501394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.501267    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:10.549188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.549146    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-639b514c-pkkzc\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.549188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.549184    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njjg\" (UniqueName: \"kubernetes.io/projected/dd07df3b-b54c-41d8-9c37-ad78d012f213-kube-api-access-4njjg\") pod \"maas-api-revoke-sub-639b514c-pkkzc\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.549388 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.549214    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-639b514c-pkkzc\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.650276 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.650243    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-639b514c-pkkzc\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.650474 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.650290    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njjg\" (UniqueName: \"kubernetes.io/projected/dd07df3b-b54c-41d8-9c37-ad78d012f213-kube-api-access-4njjg\") pod \"maas-api-revoke-sub-639b514c-pkkzc\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.650474 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.650338    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-639b514c-pkkzc\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.650946 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.650918    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-639b514c-pkkzc\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.653056 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.653033    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-639b514c-pkkzc\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.679895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.679867    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njjg\" (UniqueName: \"kubernetes.io/projected/dd07df3b-b54c-41d8-9c37-ad78d012f213-kube-api-access-4njjg\") pod \"maas-api-revoke-sub-639b514c-pkkzc\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:10.790064 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:10.790030    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:11.096199 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:11.096169    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"]
Nov 02 13:56:11.096623 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:56:11.096595    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd07df3b_b54c_41d8_9c37_ad78d012f213.slice/crio-f3b0c62656ddd59292dced6c8326cfe50dbc17d0293a155329af7f11085983a9 WatchSource:0}: Error finding container f3b0c62656ddd59292dced6c8326cfe50dbc17d0293a155329af7f11085983a9: Status 404 returned error can't find the container with id f3b0c62656ddd59292dced6c8326cfe50dbc17d0293a155329af7f11085983a9
Nov 02 13:56:11.933385 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:11.933349    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc" event={"ID":"dd07df3b-b54c-41d8-9c37-ad78d012f213","Type":"ContainerStarted","Data":"270bc9155027cce30ded72f8c4b9891985dc888f7a6e8a0e19fb028be3961fcf"}
Nov 02 13:56:11.933834 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:11.933413    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc" event={"ID":"dd07df3b-b54c-41d8-9c37-ad78d012f213","Type":"ContainerStarted","Data":"f3b0c62656ddd59292dced6c8326cfe50dbc17d0293a155329af7f11085983a9"}
Nov 02 13:56:11.983677 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:11.983621    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc" podStartSLOduration=1.9836072150000001 podStartE2EDuration="1.983607215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:11.981356925 +0000 UTC m=+762.732661365" watchObservedRunningTime="2026-11-02 13:56:11.983607215 +0000 UTC m=+762.734911650"
Nov 02 13:56:12.051048 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.049297    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"]
Nov 02 13:56:12.054840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.054165    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.076797 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.076614    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f30173-openshift-default-dockercfg-tw6wb\""
Nov 02 13:56:12.114913 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.114827    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"]
Nov 02 13:56:12.229892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.218650    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.229892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.218692    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.229892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.218720    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.229892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.218766    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.229892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.218816    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvjvp\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-kube-api-access-lvjvp\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.229892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.218847    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.229892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.218981    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.229892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.219004    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f724a8-1fa4-48c1-aace-016fd015dbac-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.229892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.219045    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.323149 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323117    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.323149 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323155    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.323397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323172    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.323397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323209    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.323397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323238    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvjvp\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-kube-api-access-lvjvp\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.323397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323269    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.323397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323381    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.323655 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323407    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f724a8-1fa4-48c1-aace-016fd015dbac-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.323655 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323439    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.324635 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.323948    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.324761 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.324694    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.324993 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.324967    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f724a8-1fa4-48c1-aace-016fd015dbac-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.325259 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.325236    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.325366 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.325340    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.327159 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.327121    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.327379 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.327361    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.350397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.350369    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.350397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.350389    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvjvp\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-kube-api-access-lvjvp\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.381680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.381653    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:12.705980 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.705953    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"]
Nov 02 13:56:12.706263 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:56:12.706237    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f724a8_1fa4_48c1_aace_016fd015dbac.slice/crio-347c8b3641c7c7510e700c27916d6b11ec16b01cf71dd400b79b9d3e0c7b5d8b WatchSource:0}: Error finding container 347c8b3641c7c7510e700c27916d6b11ec16b01cf71dd400b79b9d3e0c7b5d8b: Status 404 returned error can't find the container with id 347c8b3641c7c7510e700c27916d6b11ec16b01cf71dd400b79b9d3e0c7b5d8b
Nov 02 13:56:12.708894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.708863    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:56:12.708984 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.708926    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:56:12.708984 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.708956    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:56:12.943678 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.943642    5034 generic.go:356] "Generic (PLEG): container finished" podID="dd07df3b-b54c-41d8-9c37-ad78d012f213" containerID="270bc9155027cce30ded72f8c4b9891985dc888f7a6e8a0e19fb028be3961fcf" exitCode=0
Nov 02 13:56:12.944204 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.943722    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc" event={"ID":"dd07df3b-b54c-41d8-9c37-ad78d012f213","Type":"ContainerDied","Data":"270bc9155027cce30ded72f8c4b9891985dc888f7a6e8a0e19fb028be3961fcf"}
Nov 02 13:56:12.945344 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.945312    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" event={"ID":"90f724a8-1fa4-48c1-aace-016fd015dbac","Type":"ContainerStarted","Data":"de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08"}
Nov 02 13:56:12.945464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:12.945374    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" event={"ID":"90f724a8-1fa4-48c1-aace-016fd015dbac","Type":"ContainerStarted","Data":"347c8b3641c7c7510e700c27916d6b11ec16b01cf71dd400b79b9d3e0c7b5d8b"}
Nov 02 13:56:13.033628 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:13.033566    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podStartSLOduration=1.033551846 podStartE2EDuration="1.033551846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:13.030879324 +0000 UTC m=+763.782183762" watchObservedRunningTime="2026-11-02 13:56:13.033551846 +0000 UTC m=+763.784856281"
Nov 02 13:56:13.382888 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:13.382808    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:13.384937 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:13.384913    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:56:13.385005 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:13.384969    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 13:56:14.153380 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.153355    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:14.347665 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.347633    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd07df3b-b54c-41d8-9c37-ad78d012f213-kube-api-access-4njjg\" (UniqueName: \"kubernetes.io/projected/dd07df3b-b54c-41d8-9c37-ad78d012f213-kube-api-access-4njjg\") pod \"dd07df3b-b54c-41d8-9c37-ad78d012f213\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") "
Nov 02 13:56:14.347872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.347737    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-xks-ca\") pod \"dd07df3b-b54c-41d8-9c37-ad78d012f213\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") "
Nov 02 13:56:14.347947 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.347869    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-openshift-ca\") pod \"dd07df3b-b54c-41d8-9c37-ad78d012f213\" (UID: \"dd07df3b-b54c-41d8-9c37-ad78d012f213\") "
Nov 02 13:56:14.348243 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.348216    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-openshift-ca" pod "dd07df3b-b54c-41d8-9c37-ad78d012f213" (UID: "dd07df3b-b54c-41d8-9c37-ad78d012f213"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:14.349847 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.349820    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd07df3b-b54c-41d8-9c37-ad78d012f213-kube-api-access-4njjg" pod "dd07df3b-b54c-41d8-9c37-ad78d012f213" (UID: "dd07df3b-b54c-41d8-9c37-ad78d012f213"). InnerVolumeSpecName "kube-api-access-4njjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:14.349927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.349905    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-xks-ca" pod "dd07df3b-b54c-41d8-9c37-ad78d012f213" (UID: "dd07df3b-b54c-41d8-9c37-ad78d012f213"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:14.395801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.392245    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:56:14.395801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.392301    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 13:56:14.448914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.448885    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:14.448914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.448909    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd07df3b-b54c-41d8-9c37-ad78d012f213-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:14.448914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.448918    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njjg\" (UniqueName: \"kubernetes.io/projected/dd07df3b-b54c-41d8-9c37-ad78d012f213-kube-api-access-4njjg\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:14.958254 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.958186    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"
Nov 02 13:56:14.958254 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.958222    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc" event={"ID":"dd07df3b-b54c-41d8-9c37-ad78d012f213","Type":"ContainerDied","Data":"f3b0c62656ddd59292dced6c8326cfe50dbc17d0293a155329af7f11085983a9"}
Nov 02 13:56:14.958499 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:14.958268    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3b0c62656ddd59292dced6c8326cfe50dbc17d0293a155329af7f11085983a9"
Nov 02 13:56:15.390803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:15.390152    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:56:15.390803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:15.390211    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 13:56:16.391179 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:16.391143    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:56:16.391765 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:16.391731    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 13:56:17.386802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:17.385874    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:56:17.386802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:17.385938    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 13:56:18.383225 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:18.383190    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 02 13:56:18.383853 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:18.383820    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 02 13:56:18.419157 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:56:18.419124    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:56:19.572976 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:19.572921    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:19.573734 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:19.573706    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:19.575076 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:19.575054    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 13:56:25.408288 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.408252    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"]
Nov 02 13:56:25.409009 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.408987    5034 cpu_manager.go:414] "containerMap: removing container" podUID="dd07df3b-b54c-41d8-9c37-ad78d012f213" containerName="revoke-keys"
Nov 02 13:56:25.409009 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.409009    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd07df3b-b54c-41d8-9c37-ad78d012f213" containerName="revoke-keys"
Nov 02 13:56:25.409186 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.409135    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd07df3b-b54c-41d8-9c37-ad78d012f213" containerName="revoke-keys"
Nov 02 13:56:25.416846 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.416812    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.432357 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.432329    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"]
Nov 02 13:56:25.441807 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.441750    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:25.441921 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.441856    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:25.442114 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.442090    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:56:25.445279 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.445260    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:25.520848 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.520819    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0220407e-ltf74\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.521011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.520930    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0220407e-ltf74\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.521159 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.521141    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksckf\" (UniqueName: \"kubernetes.io/projected/d77e9d2f-ac79-421c-a99d-ef56c7749d83-kube-api-access-ksckf\") pod \"maas-api-revoke-sub-0220407e-ltf74\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.622438 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.622406    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0220407e-ltf74\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.622579 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.622457    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0220407e-ltf74\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.622579 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.622499    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksckf\" (UniqueName: \"kubernetes.io/projected/d77e9d2f-ac79-421c-a99d-ef56c7749d83-kube-api-access-ksckf\") pod \"maas-api-revoke-sub-0220407e-ltf74\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.623169 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.623150    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0220407e-ltf74\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.624792 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.624760    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0220407e-ltf74\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.646403 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.646373    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksckf\" (UniqueName: \"kubernetes.io/projected/d77e9d2f-ac79-421c-a99d-ef56c7749d83-kube-api-access-ksckf\") pod \"maas-api-revoke-sub-0220407e-ltf74\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:25.734206 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:25.734174    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:26.035236 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:26.035200    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"]
Nov 02 13:56:26.036274 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:56:26.036249    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77e9d2f_ac79_421c_a99d_ef56c7749d83.slice/crio-5a8149c761c066bf03fe9dce4b438eadf26b01a100867d8993447e5d2a3a3341 WatchSource:0}: Error finding container 5a8149c761c066bf03fe9dce4b438eadf26b01a100867d8993447e5d2a3a3341: Status 404 returned error can't find the container with id 5a8149c761c066bf03fe9dce4b438eadf26b01a100867d8993447e5d2a3a3341
Nov 02 13:56:26.068334 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:26.068304    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74" event={"ID":"d77e9d2f-ac79-421c-a99d-ef56c7749d83","Type":"ContainerStarted","Data":"5a8149c761c066bf03fe9dce4b438eadf26b01a100867d8993447e5d2a3a3341"}
Nov 02 13:56:27.088430 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:27.088378    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74" event={"ID":"d77e9d2f-ac79-421c-a99d-ef56c7749d83","Type":"ContainerStarted","Data":"6703efcad27e1855599e34b07ce40623d8c94e207dc107b073e2116db533efd7"}
Nov 02 13:56:27.130804 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:27.130438    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74" podStartSLOduration=2.130422035 podStartE2EDuration="2.130422035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:27.130136219 +0000 UTC m=+777.881440658" watchObservedRunningTime="2026-11-02 13:56:27.130422035 +0000 UTC m=+777.881726474"
Nov 02 13:56:28.098646 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:28.098613    5034 generic.go:356] "Generic (PLEG): container finished" podID="d77e9d2f-ac79-421c-a99d-ef56c7749d83" containerID="6703efcad27e1855599e34b07ce40623d8c94e207dc107b073e2116db533efd7" exitCode=0
Nov 02 13:56:28.099116 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:28.098703    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74" event={"ID":"d77e9d2f-ac79-421c-a99d-ef56c7749d83","Type":"ContainerDied","Data":"6703efcad27e1855599e34b07ce40623d8c94e207dc107b073e2116db533efd7"}
Nov 02 13:56:29.283795 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.283754    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:29.375512 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.375409    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-xks-ca\") pod \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") "
Nov 02 13:56:29.375684 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.375550    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-openshift-ca\") pod \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") "
Nov 02 13:56:29.375684 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.375606    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77e9d2f-ac79-421c-a99d-ef56c7749d83-kube-api-access-ksckf\" (UniqueName: \"kubernetes.io/projected/d77e9d2f-ac79-421c-a99d-ef56c7749d83-kube-api-access-ksckf\") pod \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\" (UID: \"d77e9d2f-ac79-421c-a99d-ef56c7749d83\") "
Nov 02 13:56:29.381833 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.376606    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-openshift-ca" pod "d77e9d2f-ac79-421c-a99d-ef56c7749d83" (UID: "d77e9d2f-ac79-421c-a99d-ef56c7749d83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:29.381833 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.380735    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77e9d2f-ac79-421c-a99d-ef56c7749d83-kube-api-access-ksckf" pod "d77e9d2f-ac79-421c-a99d-ef56c7749d83" (UID: "d77e9d2f-ac79-421c-a99d-ef56c7749d83"). InnerVolumeSpecName "kube-api-access-ksckf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:29.381833 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.381078    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-xks-ca" pod "d77e9d2f-ac79-421c-a99d-ef56c7749d83" (UID: "d77e9d2f-ac79-421c-a99d-ef56c7749d83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:29.479183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.479149    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:29.479183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.479177    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksckf\" (UniqueName: \"kubernetes.io/projected/d77e9d2f-ac79-421c-a99d-ef56c7749d83-kube-api-access-ksckf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:29.479369 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:29.479193    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77e9d2f-ac79-421c-a99d-ef56c7749d83-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:30.112216 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:30.112185    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"
Nov 02 13:56:30.112390 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:30.112190    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74" event={"ID":"d77e9d2f-ac79-421c-a99d-ef56c7749d83","Type":"ContainerDied","Data":"5a8149c761c066bf03fe9dce4b438eadf26b01a100867d8993447e5d2a3a3341"}
Nov 02 13:56:30.112390 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:30.112335    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a8149c761c066bf03fe9dce4b438eadf26b01a100867d8993447e5d2a3a3341"
Nov 02 13:56:33.485155 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.485110    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"]
Nov 02 13:56:33.485932 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.485907    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d77e9d2f-ac79-421c-a99d-ef56c7749d83" containerName="revoke-keys"
Nov 02 13:56:33.485932 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.485931    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77e9d2f-ac79-421c-a99d-ef56c7749d83" containerName="revoke-keys"
Nov 02 13:56:33.486139 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.486105    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77e9d2f-ac79-421c-a99d-ef56c7749d83" containerName="revoke-keys"
Nov 02 13:56:33.491630 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.491607    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.513999 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.513969    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"]
Nov 02 13:56:33.529045 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.529009    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:33.529200 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.529086    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:56:33.529200 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.529149    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:33.529200 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.529197    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:33.629476 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.629445    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10f9489-clqtr\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.629675 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.629488    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10f9489-clqtr\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.629675 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.629622    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5qm\" (UniqueName: \"kubernetes.io/projected/24f7df55-c680-4385-b43e-0c5a0028c664-kube-api-access-sd5qm\") pod \"maas-api-revoke-sub-e10f9489-clqtr\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.730819 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.730789    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5qm\" (UniqueName: \"kubernetes.io/projected/24f7df55-c680-4385-b43e-0c5a0028c664-kube-api-access-sd5qm\") pod \"maas-api-revoke-sub-e10f9489-clqtr\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.730992 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.730950    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10f9489-clqtr\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.730992 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.730981    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10f9489-clqtr\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.731663 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.731639    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e10f9489-clqtr\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.733511 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.733484    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e10f9489-clqtr\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.755028 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.754980    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5qm\" (UniqueName: \"kubernetes.io/projected/24f7df55-c680-4385-b43e-0c5a0028c664-kube-api-access-sd5qm\") pod \"maas-api-revoke-sub-e10f9489-clqtr\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.776801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.774461    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"]
Nov 02 13:56:33.783380 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.783362    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:33.800065 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.800042    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"]
Nov 02 13:56:33.809137 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.809116    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:33.934510 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.934477    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5e942a-z2jvk\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:33.934682 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.934585    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5e942a-z2jvk\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:33.934760 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:33.934706    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-kube-api-access-84j77\") pod \"maas-api-revoke-sub-9c5e942a-z2jvk\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:34.035726 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.035652    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-kube-api-access-84j77\") pod \"maas-api-revoke-sub-9c5e942a-z2jvk\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:34.035885 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.035834    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5e942a-z2jvk\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:34.035937 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.035900    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5e942a-z2jvk\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:34.036474 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.036452    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5e942a-z2jvk\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:34.038179 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.038157    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5e942a-z2jvk\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:34.059168 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.059148    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-kube-api-access-84j77\") pod \"maas-api-revoke-sub-9c5e942a-z2jvk\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:34.102030 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.101801    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:34.144051 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.144025    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"]
Nov 02 13:56:34.145471 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:56:34.145439    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f7df55_c680_4385_b43e_0c5a0028c664.slice/crio-93ce80f83c23e2fe1d517112102ccacfc42c59ab4f96233adc16c1ba3b8257f5 WatchSource:0}: Error finding container 93ce80f83c23e2fe1d517112102ccacfc42c59ab4f96233adc16c1ba3b8257f5: Status 404 returned error can't find the container with id 93ce80f83c23e2fe1d517112102ccacfc42c59ab4f96233adc16c1ba3b8257f5
Nov 02 13:56:34.413812 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:56:34.413116    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a020e3b_3fd5_4edb_bd21_a2dbe5f99773.slice/crio-77c9bfbfca9eec442af3195edddece72161c0ccf74e7bf922e81b1d498ed311c WatchSource:0}: Error finding container 77c9bfbfca9eec442af3195edddece72161c0ccf74e7bf922e81b1d498ed311c: Status 404 returned error can't find the container with id 77c9bfbfca9eec442af3195edddece72161c0ccf74e7bf922e81b1d498ed311c
Nov 02 13:56:34.413812 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.413689    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"]
Nov 02 13:56:34.802143 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.802110    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"]
Nov 02 13:56:34.806209 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.806187    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:34.836198 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.836178    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"]
Nov 02 13:56:34.946397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.946316    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bbf4923-6t6ln\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:34.946397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.946374    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bbf4923-6t6ln\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:34.946575 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:34.946487    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg897\" (UniqueName: \"kubernetes.io/projected/9dc14d42-c40b-45ee-950f-9fe330b69256-kube-api-access-dg897\") pod \"maas-api-revoke-sub-4bbf4923-6t6ln\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:35.048073 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.048041    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bbf4923-6t6ln\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:35.048218 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.048118    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg897\" (UniqueName: \"kubernetes.io/projected/9dc14d42-c40b-45ee-950f-9fe330b69256-kube-api-access-dg897\") pod \"maas-api-revoke-sub-4bbf4923-6t6ln\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:35.048288 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.048223    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bbf4923-6t6ln\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:35.048828 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.048803    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bbf4923-6t6ln\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:35.050430 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.050408    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bbf4923-6t6ln\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:35.073730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.073702    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg897\" (UniqueName: \"kubernetes.io/projected/9dc14d42-c40b-45ee-950f-9fe330b69256-kube-api-access-dg897\") pod \"maas-api-revoke-sub-4bbf4923-6t6ln\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:35.120291 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.120261    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:35.144046 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.144012    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr" event={"ID":"24f7df55-c680-4385-b43e-0c5a0028c664","Type":"ContainerStarted","Data":"4e446fa3ea09298b0c8ff047207feaa4af8d46f94968edddfee0a0d4198f3911"}
Nov 02 13:56:35.144163 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.144080    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr" event={"ID":"24f7df55-c680-4385-b43e-0c5a0028c664","Type":"ContainerStarted","Data":"93ce80f83c23e2fe1d517112102ccacfc42c59ab4f96233adc16c1ba3b8257f5"}
Nov 02 13:56:35.145834 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.145801    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk" event={"ID":"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773","Type":"ContainerStarted","Data":"7d093c6eb77a23442e6aec5b250bcfd2e48cfa8bb86d12988ff78b95e6017a0b"}
Nov 02 13:56:35.145981 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.145854    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk" event={"ID":"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773","Type":"ContainerStarted","Data":"77c9bfbfca9eec442af3195edddece72161c0ccf74e7bf922e81b1d498ed311c"}
Nov 02 13:56:35.212601 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.212494    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr" podStartSLOduration=2.212472959 podStartE2EDuration="2.212472959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:35.21097459 +0000 UTC m=+785.962279030" watchObservedRunningTime="2026-11-02 13:56:35.212472959 +0000 UTC m=+785.963777399"
Nov 02 13:56:35.281799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.279259    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk" podStartSLOduration=2.279237383 podStartE2EDuration="2.279237383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:35.2784613 +0000 UTC m=+786.029765742" watchObservedRunningTime="2026-11-02 13:56:35.279237383 +0000 UTC m=+786.030541822"
Nov 02 13:56:35.679994 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:56:35.679961    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc14d42_c40b_45ee_950f_9fe330b69256.slice/crio-eae362a870175a74ff78bb14330340e51b97822c680b4da4e68707759e02f9fe WatchSource:0}: Error finding container eae362a870175a74ff78bb14330340e51b97822c680b4da4e68707759e02f9fe: Status 404 returned error can't find the container with id eae362a870175a74ff78bb14330340e51b97822c680b4da4e68707759e02f9fe
Nov 02 13:56:35.681444 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:35.681416    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"]
Nov 02 13:56:36.162860 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:36.162815    5034 generic.go:356] "Generic (PLEG): container finished" podID="4a020e3b-3fd5-4edb-bd21-a2dbe5f99773" containerID="7d093c6eb77a23442e6aec5b250bcfd2e48cfa8bb86d12988ff78b95e6017a0b" exitCode=0
Nov 02 13:56:36.163354 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:36.162869    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk" event={"ID":"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773","Type":"ContainerDied","Data":"7d093c6eb77a23442e6aec5b250bcfd2e48cfa8bb86d12988ff78b95e6017a0b"}
Nov 02 13:56:36.165084 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:36.165052    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln" event={"ID":"9dc14d42-c40b-45ee-950f-9fe330b69256","Type":"ContainerStarted","Data":"0599dde52c78d81f0c28b756899ad0868362f53c5445f818b22c5dedba559052"}
Nov 02 13:56:36.165220 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:36.165105    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln" event={"ID":"9dc14d42-c40b-45ee-950f-9fe330b69256","Type":"ContainerStarted","Data":"eae362a870175a74ff78bb14330340e51b97822c680b4da4e68707759e02f9fe"}
Nov 02 13:56:36.166843 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:36.166821    5034 generic.go:356] "Generic (PLEG): container finished" podID="24f7df55-c680-4385-b43e-0c5a0028c664" containerID="4e446fa3ea09298b0c8ff047207feaa4af8d46f94968edddfee0a0d4198f3911" exitCode=0
Nov 02 13:56:36.166982 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:36.166869    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr" event={"ID":"24f7df55-c680-4385-b43e-0c5a0028c664","Type":"ContainerDied","Data":"4e446fa3ea09298b0c8ff047207feaa4af8d46f94968edddfee0a0d4198f3911"}
Nov 02 13:56:36.329845 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:36.323528    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln" podStartSLOduration=2.32350942 podStartE2EDuration="2.32350942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:36.261991554 +0000 UTC m=+787.013296023" watchObservedRunningTime="2026-11-02 13:56:36.32350942 +0000 UTC m=+787.074813861"
Nov 02 13:56:37.174619 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.174579    5034 generic.go:356] "Generic (PLEG): container finished" podID="9dc14d42-c40b-45ee-950f-9fe330b69256" containerID="0599dde52c78d81f0c28b756899ad0868362f53c5445f818b22c5dedba559052" exitCode=0
Nov 02 13:56:37.175102 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.174660    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln" event={"ID":"9dc14d42-c40b-45ee-950f-9fe330b69256","Type":"ContainerDied","Data":"0599dde52c78d81f0c28b756899ad0868362f53c5445f818b22c5dedba559052"}
Nov 02 13:56:37.394978 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.394954    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:37.430024 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.429966    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:37.435766 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.435747    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f7df55-c680-4385-b43e-0c5a0028c664-kube-api-access-sd5qm\" (UniqueName: \"kubernetes.io/projected/24f7df55-c680-4385-b43e-0c5a0028c664-kube-api-access-sd5qm\") pod \"24f7df55-c680-4385-b43e-0c5a0028c664\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") "
Nov 02 13:56:37.435894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.435796    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-kube-api-access-84j77\") pod \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") "
Nov 02 13:56:37.435894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.435835    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-xks-ca\") pod \"24f7df55-c680-4385-b43e-0c5a0028c664\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") "
Nov 02 13:56:37.436007 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.435934    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-openshift-ca\") pod \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") "
Nov 02 13:56:37.436007 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.435976    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-openshift-ca\") pod \"24f7df55-c680-4385-b43e-0c5a0028c664\" (UID: \"24f7df55-c680-4385-b43e-0c5a0028c664\") "
Nov 02 13:56:37.436115 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.436013    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-xks-ca\") pod \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\" (UID: \"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773\") "
Nov 02 13:56:37.436380 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.436355    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-openshift-ca" pod "4a020e3b-3fd5-4edb-bd21-a2dbe5f99773" (UID: "4a020e3b-3fd5-4edb-bd21-a2dbe5f99773"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:37.436467 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.436356    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-openshift-ca" pod "24f7df55-c680-4385-b43e-0c5a0028c664" (UID: "24f7df55-c680-4385-b43e-0c5a0028c664"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:37.436529 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.436513    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:37.436590 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.436533    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:37.438149 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.438125    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f7df55-c680-4385-b43e-0c5a0028c664-kube-api-access-sd5qm" pod "24f7df55-c680-4385-b43e-0c5a0028c664" (UID: "24f7df55-c680-4385-b43e-0c5a0028c664"). InnerVolumeSpecName "kube-api-access-sd5qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:37.438367 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.438311    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-xks-ca" pod "24f7df55-c680-4385-b43e-0c5a0028c664" (UID: "24f7df55-c680-4385-b43e-0c5a0028c664"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:37.439655 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.439628    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-xks-ca" pod "4a020e3b-3fd5-4edb-bd21-a2dbe5f99773" (UID: "4a020e3b-3fd5-4edb-bd21-a2dbe5f99773"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:37.439813 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.439671    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-kube-api-access-84j77" pod "4a020e3b-3fd5-4edb-bd21-a2dbe5f99773" (UID: "4a020e3b-3fd5-4edb-bd21-a2dbe5f99773"). InnerVolumeSpecName "kube-api-access-84j77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:37.537506 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.537476    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:37.537506 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.537501    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd5qm\" (UniqueName: \"kubernetes.io/projected/24f7df55-c680-4385-b43e-0c5a0028c664-kube-api-access-sd5qm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:37.537506 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.537511    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84j77\" (UniqueName: \"kubernetes.io/projected/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773-kube-api-access-84j77\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:37.537705 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:37.537521    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f7df55-c680-4385-b43e-0c5a0028c664-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:38.183468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.183436    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"
Nov 02 13:56:38.183468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.183463    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk" event={"ID":"4a020e3b-3fd5-4edb-bd21-a2dbe5f99773","Type":"ContainerDied","Data":"77c9bfbfca9eec442af3195edddece72161c0ccf74e7bf922e81b1d498ed311c"}
Nov 02 13:56:38.184029 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.183518    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77c9bfbfca9eec442af3195edddece72161c0ccf74e7bf922e81b1d498ed311c"
Nov 02 13:56:38.185148 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.185117    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"
Nov 02 13:56:38.185288 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.185174    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr" event={"ID":"24f7df55-c680-4385-b43e-0c5a0028c664","Type":"ContainerDied","Data":"93ce80f83c23e2fe1d517112102ccacfc42c59ab4f96233adc16c1ba3b8257f5"}
Nov 02 13:56:38.185288 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.185209    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93ce80f83c23e2fe1d517112102ccacfc42c59ab4f96233adc16c1ba3b8257f5"
Nov 02 13:56:38.322589 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.322564    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:38.345049 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.345028    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-xks-ca\") pod \"9dc14d42-c40b-45ee-950f-9fe330b69256\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") "
Nov 02 13:56:38.345165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.345076    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc14d42-c40b-45ee-950f-9fe330b69256-kube-api-access-dg897\" (UniqueName: \"kubernetes.io/projected/9dc14d42-c40b-45ee-950f-9fe330b69256-kube-api-access-dg897\") pod \"9dc14d42-c40b-45ee-950f-9fe330b69256\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") "
Nov 02 13:56:38.345165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.345138    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-openshift-ca\") pod \"9dc14d42-c40b-45ee-950f-9fe330b69256\" (UID: \"9dc14d42-c40b-45ee-950f-9fe330b69256\") "
Nov 02 13:56:38.345540 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.345519    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-openshift-ca" pod "9dc14d42-c40b-45ee-950f-9fe330b69256" (UID: "9dc14d42-c40b-45ee-950f-9fe330b69256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:38.345679 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.345662    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:38.347158 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.347130    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-xks-ca" pod "9dc14d42-c40b-45ee-950f-9fe330b69256" (UID: "9dc14d42-c40b-45ee-950f-9fe330b69256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:38.347233 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.347192    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc14d42-c40b-45ee-950f-9fe330b69256-kube-api-access-dg897" pod "9dc14d42-c40b-45ee-950f-9fe330b69256" (UID: "9dc14d42-c40b-45ee-950f-9fe330b69256"). InnerVolumeSpecName "kube-api-access-dg897". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:38.446606 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.446542    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc14d42-c40b-45ee-950f-9fe330b69256-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:38.446606 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:38.446567    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg897\" (UniqueName: \"kubernetes.io/projected/9dc14d42-c40b-45ee-950f-9fe330b69256-kube-api-access-dg897\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:39.193442 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:39.193404    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln" event={"ID":"9dc14d42-c40b-45ee-950f-9fe330b69256","Type":"ContainerDied","Data":"eae362a870175a74ff78bb14330340e51b97822c680b4da4e68707759e02f9fe"}
Nov 02 13:56:39.193442 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:39.193439    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"
Nov 02 13:56:39.193914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:39.193462    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae362a870175a74ff78bb14330340e51b97822c680b4da4e68707759e02f9fe"
Nov 02 13:56:43.150303 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150270    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"]
Nov 02 13:56:43.150768 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150755    5034 cpu_manager.go:414] "containerMap: removing container" podUID="24f7df55-c680-4385-b43e-0c5a0028c664" containerName="revoke-keys"
Nov 02 13:56:43.150851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150769    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f7df55-c680-4385-b43e-0c5a0028c664" containerName="revoke-keys"
Nov 02 13:56:43.150851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150801    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4a020e3b-3fd5-4edb-bd21-a2dbe5f99773" containerName="revoke-keys"
Nov 02 13:56:43.150851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150806    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a020e3b-3fd5-4edb-bd21-a2dbe5f99773" containerName="revoke-keys"
Nov 02 13:56:43.150851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150815    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9dc14d42-c40b-45ee-950f-9fe330b69256" containerName="revoke-keys"
Nov 02 13:56:43.150851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150821    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc14d42-c40b-45ee-950f-9fe330b69256" containerName="revoke-keys"
Nov 02 13:56:43.150995 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150927    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f7df55-c680-4385-b43e-0c5a0028c664" containerName="revoke-keys"
Nov 02 13:56:43.150995 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150936    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a020e3b-3fd5-4edb-bd21-a2dbe5f99773" containerName="revoke-keys"
Nov 02 13:56:43.150995 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.150943    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc14d42-c40b-45ee-950f-9fe330b69256" containerName="revoke-keys"
Nov 02 13:56:43.154553 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.154533    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.169941 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.169921    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-ad42e0-openshift-default-dockercfg-gmvz9\""
Nov 02 13:56:43.186137 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.186113    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"]
Nov 02 13:56:43.195818 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.195012    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.195818 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.195062    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.195818 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.195143    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.195818 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.195169    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.195818 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.195248    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.195818 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.195335    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.195818 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.195376    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.195818 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.195495    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h7mt\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-kube-api-access-9h7mt\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.195818 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.195535    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315df3e6-f69e-469f-b78f-ad9bae83a00d-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.296928 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.296901    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h7mt\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-kube-api-access-9h7mt\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297058 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.296934    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315df3e6-f69e-469f-b78f-ad9bae83a00d-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297058 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.296962    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297058 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.296985    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297251 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297121    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297251 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297167    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297251 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297215    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297407 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297302    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297407 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297338    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297407 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297388    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297522    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297755 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297653    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297870 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297773    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315df3e6-f69e-469f-b78f-ad9bae83a00d-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.297950 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.297894    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.299584 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.299561    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.299995 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.299976    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.320761 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.320727    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.320880 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.320865    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h7mt\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-kube-api-access-9h7mt\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.467688 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.467606    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:43.798407 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.798384    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"]
Nov 02 13:56:43.799416 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:56:43.799388    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod315df3e6_f69e_469f_b78f_ad9bae83a00d.slice/crio-7e3f6b20823de0e33983e82c335889ba655b51b3f2c144a3fe928b1c9062bc00 WatchSource:0}: Error finding container 7e3f6b20823de0e33983e82c335889ba655b51b3f2c144a3fe928b1c9062bc00: Status 404 returned error can't find the container with id 7e3f6b20823de0e33983e82c335889ba655b51b3f2c144a3fe928b1c9062bc00
Nov 02 13:56:43.802438 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.802405    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:56:43.802535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.802482    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:56:43.802535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:43.802528    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:56:44.228586 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:44.228493    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" event={"ID":"315df3e6-f69e-469f-b78f-ad9bae83a00d","Type":"ContainerStarted","Data":"09cd10d595a3384caf9ef9b067164087fd7f753f1ee8f87e8881751f8330043c"}
Nov 02 13:56:44.228586 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:44.228546    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" event={"ID":"315df3e6-f69e-469f-b78f-ad9bae83a00d","Type":"ContainerStarted","Data":"7e3f6b20823de0e33983e82c335889ba655b51b3f2c144a3fe928b1c9062bc00"}
Nov 02 13:56:44.288880 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:44.288823    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" podStartSLOduration=1.288808398 podStartE2EDuration="1.288808398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:44.286111522 +0000 UTC m=+795.037415970" watchObservedRunningTime="2026-11-02 13:56:44.288808398 +0000 UTC m=+795.040112835"
Nov 02 13:56:44.468319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:44.468289    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:44.473801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:44.472882    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:56:44.473801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:44.472952    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 13:56:45.468252 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:45.468215    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:56:45.468755 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:45.468279    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 13:56:46.468120 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:46.468081    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:56:46.468323 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:46.468156    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 13:56:47.468538 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:47.468507    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:56:47.469016 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:47.468565    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 13:56:48.424136 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:56:48.424096    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:56:48.468332 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:48.468307    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:56:48.468466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:48.468364    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 13:56:48.966488 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:48.966455    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"]
Nov 02 13:56:48.986800 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:48.983152    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:48.999864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:48.999831    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"]
Nov 02 13:56:49.004388 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.004343    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:56:49.016918 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.016891    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:49.016918 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.016911    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:56:49.017059 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.016987    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:56:49.106892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.106854    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523509a0-g62pk\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.107083 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.106918    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nbb\" (UniqueName: \"kubernetes.io/projected/3db1517b-dbe2-4d18-b155-a87740ea7f75-kube-api-access-x5nbb\") pod \"maas-api-revoke-sub-523509a0-g62pk\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.107083 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.107055    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523509a0-g62pk\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.208163 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.208129    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523509a0-g62pk\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.208360 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.208275    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523509a0-g62pk\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.208360 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.208322    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nbb\" (UniqueName: \"kubernetes.io/projected/3db1517b-dbe2-4d18-b155-a87740ea7f75-kube-api-access-x5nbb\") pod \"maas-api-revoke-sub-523509a0-g62pk\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.209006 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.208972    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-523509a0-g62pk\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.210650 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.210630    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-523509a0-g62pk\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.233806 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.233768    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nbb\" (UniqueName: \"kubernetes.io/projected/3db1517b-dbe2-4d18-b155-a87740ea7f75-kube-api-access-x5nbb\") pod \"maas-api-revoke-sub-523509a0-g62pk\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.325976 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.316986    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:49.493991 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.493896    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Nov 02 13:56:49.493991 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.493957    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Nov 02 13:56:49.664944 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:49.664844    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"]
Nov 02 13:56:49.665131 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:56:49.665103    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db1517b_dbe2_4d18_b155_a87740ea7f75.slice/crio-3bb8c913fe7271c834f91b8fef30f5ae7803c76262f979ee3e687ee06ef877d0 WatchSource:0}: Error finding container 3bb8c913fe7271c834f91b8fef30f5ae7803c76262f979ee3e687ee06ef877d0: Status 404 returned error can't find the container with id 3bb8c913fe7271c834f91b8fef30f5ae7803c76262f979ee3e687ee06ef877d0
Nov 02 13:56:50.271693 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:50.271592    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk" event={"ID":"3db1517b-dbe2-4d18-b155-a87740ea7f75","Type":"ContainerStarted","Data":"8768532411feb40c2bccff9f7d63b79d3fedff527c8cdcc25840eebdfedbfeaa"}
Nov 02 13:56:50.272186 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:50.271708    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk" event={"ID":"3db1517b-dbe2-4d18-b155-a87740ea7f75","Type":"ContainerStarted","Data":"3bb8c913fe7271c834f91b8fef30f5ae7803c76262f979ee3e687ee06ef877d0"}
Nov 02 13:56:50.314496 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:50.314430    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk" podStartSLOduration=2.314409664 podStartE2EDuration="2.314409664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:50.312423645 +0000 UTC m=+801.063728088" watchObservedRunningTime="2026-11-02 13:56:50.314409664 +0000 UTC m=+801.065714103"
Nov 02 13:56:50.595254 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:50.595171    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:50.596123 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:50.596097    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:50.596838 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:50.596818    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 13:56:51.280871 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:51.280833    5034 generic.go:356] "Generic (PLEG): container finished" podID="3db1517b-dbe2-4d18-b155-a87740ea7f75" containerID="8768532411feb40c2bccff9f7d63b79d3fedff527c8cdcc25840eebdfedbfeaa" exitCode=0
Nov 02 13:56:51.281827 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:51.281801    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk" event={"ID":"3db1517b-dbe2-4d18-b155-a87740ea7f75","Type":"ContainerDied","Data":"8768532411feb40c2bccff9f7d63b79d3fedff527c8cdcc25840eebdfedbfeaa"}
Nov 02 13:56:52.494580 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.494551    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:52.664916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.664842    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db1517b-dbe2-4d18-b155-a87740ea7f75-kube-api-access-x5nbb\" (UniqueName: \"kubernetes.io/projected/3db1517b-dbe2-4d18-b155-a87740ea7f75-kube-api-access-x5nbb\") pod \"3db1517b-dbe2-4d18-b155-a87740ea7f75\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") "
Nov 02 13:56:52.664916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.664905    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-xks-ca\") pod \"3db1517b-dbe2-4d18-b155-a87740ea7f75\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") "
Nov 02 13:56:52.665081 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.664975    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-openshift-ca\") pod \"3db1517b-dbe2-4d18-b155-a87740ea7f75\" (UID: \"3db1517b-dbe2-4d18-b155-a87740ea7f75\") "
Nov 02 13:56:52.665320 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.665295    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-openshift-ca" pod "3db1517b-dbe2-4d18-b155-a87740ea7f75" (UID: "3db1517b-dbe2-4d18-b155-a87740ea7f75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:52.667004 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.666983    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db1517b-dbe2-4d18-b155-a87740ea7f75-kube-api-access-x5nbb" pod "3db1517b-dbe2-4d18-b155-a87740ea7f75" (UID: "3db1517b-dbe2-4d18-b155-a87740ea7f75"). InnerVolumeSpecName "kube-api-access-x5nbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:52.667101 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.667045    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-xks-ca" pod "3db1517b-dbe2-4d18-b155-a87740ea7f75" (UID: "3db1517b-dbe2-4d18-b155-a87740ea7f75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:52.766378 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.766350    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:52.766378 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.766374    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5nbb\" (UniqueName: \"kubernetes.io/projected/3db1517b-dbe2-4d18-b155-a87740ea7f75-kube-api-access-x5nbb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:52.766378 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:52.766384    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3db1517b-dbe2-4d18-b155-a87740ea7f75-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:56:53.299162 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:53.299134    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"
Nov 02 13:56:53.299334 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:53.299185    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk" event={"ID":"3db1517b-dbe2-4d18-b155-a87740ea7f75","Type":"ContainerDied","Data":"3bb8c913fe7271c834f91b8fef30f5ae7803c76262f979ee3e687ee06ef877d0"}
Nov 02 13:56:53.299334 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:56:53.299243    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bb8c913fe7271c834f91b8fef30f5ae7803c76262f979ee3e687ee06ef877d0"
Nov 02 13:57:01.279117 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.278928    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"]
Nov 02 13:57:01.281815 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.279749    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3db1517b-dbe2-4d18-b155-a87740ea7f75" containerName="revoke-keys"
Nov 02 13:57:01.281815 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.279772    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db1517b-dbe2-4d18-b155-a87740ea7f75" containerName="revoke-keys"
Nov 02 13:57:01.281815 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.279987    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db1517b-dbe2-4d18-b155-a87740ea7f75" containerName="revoke-keys"
Nov 02 13:57:01.294169 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.294130    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.305270 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.305240    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"]
Nov 02 13:57:01.314090 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.314066    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:57:01.315085 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.315052    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:57:01.315392 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.315366    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:01.322757 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.322732    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:57:01.324558 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.324531    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3346fc4-6kxp4\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.324672 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.324655    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j2l8\" (UniqueName: \"kubernetes.io/projected/311ad3eb-5c96-4aac-a482-649df208e07b-kube-api-access-2j2l8\") pod \"maas-api-revoke-sub-b3346fc4-6kxp4\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.324749 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.324728    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3346fc4-6kxp4\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.425697 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.425668    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j2l8\" (UniqueName: \"kubernetes.io/projected/311ad3eb-5c96-4aac-a482-649df208e07b-kube-api-access-2j2l8\") pod \"maas-api-revoke-sub-b3346fc4-6kxp4\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.425830 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.425721    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3346fc4-6kxp4\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.425871 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.425835    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3346fc4-6kxp4\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.426513 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.426491    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3346fc4-6kxp4\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.428107 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.428081    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3346fc4-6kxp4\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.449512 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.449481    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j2l8\" (UniqueName: \"kubernetes.io/projected/311ad3eb-5c96-4aac-a482-649df208e07b-kube-api-access-2j2l8\") pod \"maas-api-revoke-sub-b3346fc4-6kxp4\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.619526 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.619459    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:01.921877 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:57:01.921846    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311ad3eb_5c96_4aac_a482_649df208e07b.slice/crio-22bbd86bcf069247affe033480f290b69cede0448a5bce10e12888a24d5edefa WatchSource:0}: Error finding container 22bbd86bcf069247affe033480f290b69cede0448a5bce10e12888a24d5edefa: Status 404 returned error can't find the container with id 22bbd86bcf069247affe033480f290b69cede0448a5bce10e12888a24d5edefa
Nov 02 13:57:01.922982 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:01.922954    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"]
Nov 02 13:57:02.404356 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:02.404321    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4" event={"ID":"311ad3eb-5c96-4aac-a482-649df208e07b","Type":"ContainerStarted","Data":"6e4e213f78417c36a7b1defb2aca4de23a9283d089402b22ff2d4ff900b5cad4"}
Nov 02 13:57:02.404732 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:02.404372    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4" event={"ID":"311ad3eb-5c96-4aac-a482-649df208e07b","Type":"ContainerStarted","Data":"22bbd86bcf069247affe033480f290b69cede0448a5bce10e12888a24d5edefa"}
Nov 02 13:57:02.449884 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:02.449798    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4" podStartSLOduration=1.449772323 podStartE2EDuration="1.449772323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:02.445815978 +0000 UTC m=+813.197120416" watchObservedRunningTime="2026-11-02 13:57:02.449772323 +0000 UTC m=+813.201076760"
Nov 02 13:57:03.411744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:03.411712    5034 generic.go:356] "Generic (PLEG): container finished" podID="311ad3eb-5c96-4aac-a482-649df208e07b" containerID="6e4e213f78417c36a7b1defb2aca4de23a9283d089402b22ff2d4ff900b5cad4" exitCode=0
Nov 02 13:57:03.412122 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:03.411813    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4" event={"ID":"311ad3eb-5c96-4aac-a482-649df208e07b","Type":"ContainerDied","Data":"6e4e213f78417c36a7b1defb2aca4de23a9283d089402b22ff2d4ff900b5cad4"}
Nov 02 13:57:04.571754 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.571727    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:04.672461 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.672384    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-xks-ca\") pod \"311ad3eb-5c96-4aac-a482-649df208e07b\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") "
Nov 02 13:57:04.672461 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.672457    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-openshift-ca\") pod \"311ad3eb-5c96-4aac-a482-649df208e07b\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") "
Nov 02 13:57:04.672680 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.672487    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/311ad3eb-5c96-4aac-a482-649df208e07b-kube-api-access-2j2l8\" (UniqueName: \"kubernetes.io/projected/311ad3eb-5c96-4aac-a482-649df208e07b-kube-api-access-2j2l8\") pod \"311ad3eb-5c96-4aac-a482-649df208e07b\" (UID: \"311ad3eb-5c96-4aac-a482-649df208e07b\") "
Nov 02 13:57:04.672901 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.672874    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-openshift-ca" pod "311ad3eb-5c96-4aac-a482-649df208e07b" (UID: "311ad3eb-5c96-4aac-a482-649df208e07b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:04.673117 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.673093    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:04.674741 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.674714    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311ad3eb-5c96-4aac-a482-649df208e07b-kube-api-access-2j2l8" pod "311ad3eb-5c96-4aac-a482-649df208e07b" (UID: "311ad3eb-5c96-4aac-a482-649df208e07b"). InnerVolumeSpecName "kube-api-access-2j2l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:04.675064 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.675046    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-xks-ca" pod "311ad3eb-5c96-4aac-a482-649df208e07b" (UID: "311ad3eb-5c96-4aac-a482-649df208e07b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:04.774660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.774635    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/311ad3eb-5c96-4aac-a482-649df208e07b-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:04.774660 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:04.774657    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j2l8\" (UniqueName: \"kubernetes.io/projected/311ad3eb-5c96-4aac-a482-649df208e07b-kube-api-access-2j2l8\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:05.427199 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:05.427175    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"
Nov 02 13:57:05.427351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:05.427203    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4" event={"ID":"311ad3eb-5c96-4aac-a482-649df208e07b","Type":"ContainerDied","Data":"22bbd86bcf069247affe033480f290b69cede0448a5bce10e12888a24d5edefa"}
Nov 02 13:57:05.427351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:05.427251    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22bbd86bcf069247affe033480f290b69cede0448a5bce10e12888a24d5edefa"
Nov 02 13:57:11.633514 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.633481    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"]
Nov 02 13:57:11.634083 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.634065    5034 cpu_manager.go:414] "containerMap: removing container" podUID="311ad3eb-5c96-4aac-a482-649df208e07b" containerName="revoke-keys"
Nov 02 13:57:11.634135 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.634084    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="311ad3eb-5c96-4aac-a482-649df208e07b" containerName="revoke-keys"
Nov 02 13:57:11.634205 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.634194    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="311ad3eb-5c96-4aac-a482-649df208e07b" containerName="revoke-keys"
Nov 02 13:57:11.638764 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.638744    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.674010 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.673985    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:11.674161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.673983    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:57:11.674161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.674049    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:57:11.674161 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.674131    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:57:11.678810 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.678763    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"]
Nov 02 13:57:11.710017 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.709991    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d660a456-9jc8l\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.710131 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.710033    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4nn4\" (UniqueName: \"kubernetes.io/projected/f8e10c7f-4197-4922-b12c-fd7bbb573aed-kube-api-access-d4nn4\") pod \"maas-api-revoke-sub-d660a456-9jc8l\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.710187 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.710171    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d660a456-9jc8l\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.811701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.811672    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d660a456-9jc8l\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.811701 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.811716    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4nn4\" (UniqueName: \"kubernetes.io/projected/f8e10c7f-4197-4922-b12c-fd7bbb573aed-kube-api-access-d4nn4\") pod \"maas-api-revoke-sub-d660a456-9jc8l\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.811935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.811800    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d660a456-9jc8l\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.812327 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.812303    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d660a456-9jc8l\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.814143 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.814119    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d660a456-9jc8l\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.836442 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.836409    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4nn4\" (UniqueName: \"kubernetes.io/projected/f8e10c7f-4197-4922-b12c-fd7bbb573aed-kube-api-access-d4nn4\") pod \"maas-api-revoke-sub-d660a456-9jc8l\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:11.951581 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:11.951504    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:12.253498 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:12.253463    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"]
Nov 02 13:57:12.254584 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:57:12.254556    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e10c7f_4197_4922_b12c_fd7bbb573aed.slice/crio-b7ff8dace0dd8b512c5d78bbe3c43054d5a02ab1f0ae4b4cffa66468c2ec5e85 WatchSource:0}: Error finding container b7ff8dace0dd8b512c5d78bbe3c43054d5a02ab1f0ae4b4cffa66468c2ec5e85: Status 404 returned error can't find the container with id b7ff8dace0dd8b512c5d78bbe3c43054d5a02ab1f0ae4b4cffa66468c2ec5e85
Nov 02 13:57:12.481645 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:12.481616    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l" event={"ID":"f8e10c7f-4197-4922-b12c-fd7bbb573aed","Type":"ContainerStarted","Data":"b7ff8dace0dd8b512c5d78bbe3c43054d5a02ab1f0ae4b4cffa66468c2ec5e85"}
Nov 02 13:57:13.489852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:13.489822    5034 generic.go:356] "Generic (PLEG): container finished" podID="f8e10c7f-4197-4922-b12c-fd7bbb573aed" containerID="88301783dac8364bbd3b6e03b23d1254e59aadac2198000261db0a62f42f3943" exitCode=0
Nov 02 13:57:13.490266 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:13.489913    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l" event={"ID":"f8e10c7f-4197-4922-b12c-fd7bbb573aed","Type":"ContainerDied","Data":"88301783dac8364bbd3b6e03b23d1254e59aadac2198000261db0a62f42f3943"}
Nov 02 13:57:14.673050 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.673028    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:14.853914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.853827    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e10c7f-4197-4922-b12c-fd7bbb573aed-kube-api-access-d4nn4\" (UniqueName: \"kubernetes.io/projected/f8e10c7f-4197-4922-b12c-fd7bbb573aed-kube-api-access-d4nn4\") pod \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") "
Nov 02 13:57:14.853914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.853907    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-openshift-ca\") pod \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") "
Nov 02 13:57:14.854124 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.854075    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-xks-ca\") pod \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\" (UID: \"f8e10c7f-4197-4922-b12c-fd7bbb573aed\") "
Nov 02 13:57:14.854286 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.854262    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-openshift-ca" pod "f8e10c7f-4197-4922-b12c-fd7bbb573aed" (UID: "f8e10c7f-4197-4922-b12c-fd7bbb573aed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:14.854562 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.854544    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:14.855903 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.855882    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e10c7f-4197-4922-b12c-fd7bbb573aed-kube-api-access-d4nn4" pod "f8e10c7f-4197-4922-b12c-fd7bbb573aed" (UID: "f8e10c7f-4197-4922-b12c-fd7bbb573aed"). InnerVolumeSpecName "kube-api-access-d4nn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:14.856059 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.856046    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-xks-ca" pod "f8e10c7f-4197-4922-b12c-fd7bbb573aed" (UID: "f8e10c7f-4197-4922-b12c-fd7bbb573aed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:14.956078 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.956046    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e10c7f-4197-4922-b12c-fd7bbb573aed-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:14.956078 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:14.956080    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4nn4\" (UniqueName: \"kubernetes.io/projected/f8e10c7f-4197-4922-b12c-fd7bbb573aed-kube-api-access-d4nn4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:15.505931 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:15.505897    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l" event={"ID":"f8e10c7f-4197-4922-b12c-fd7bbb573aed","Type":"ContainerDied","Data":"b7ff8dace0dd8b512c5d78bbe3c43054d5a02ab1f0ae4b4cffa66468c2ec5e85"}
Nov 02 13:57:15.505931 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:15.505922    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"
Nov 02 13:57:15.506131 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:15.505944    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7ff8dace0dd8b512c5d78bbe3c43054d5a02ab1f0ae4b4cffa66468c2ec5e85"
Nov 02 13:57:16.056799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.052854    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"]
Nov 02 13:57:16.056799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.053770    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f8e10c7f-4197-4922-b12c-fd7bbb573aed" containerName="revoke-keys"
Nov 02 13:57:16.056799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.053808    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e10c7f-4197-4922-b12c-fd7bbb573aed" containerName="revoke-keys"
Nov 02 13:57:16.056799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.053944    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e10c7f-4197-4922-b12c-fd7bbb573aed" containerName="revoke-keys"
Nov 02 13:57:16.065101 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.065070    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.113021 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.112988    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"]
Nov 02 13:57:16.174950 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.174910    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.175112 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.174956    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.175112 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.175057    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb41e2-890d-4add-86b5-39d8581198d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.175248 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.175169    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.175248 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.175231    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.175351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.175325    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.175405 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.175356    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.175405 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.175388    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z762t\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-kube-api-access-z762t\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.175520 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.175471    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.232400 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.232371    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"]
Nov 02 13:57:16.236315 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.236298    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.257050 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.257025    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"]
Nov 02 13:57:16.258017 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.257997    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:16.258166 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.258096    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:57:16.258229 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.258111    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:57:16.258276 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.258236    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:57:16.277062 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.277031    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.277188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.277127    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.277188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.277164    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.277287 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.277201    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.277287 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.277242    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z762t\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-kube-api-access-z762t\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.277287 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.277279    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.277406 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.277364    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.277406 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.277392    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.277475 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.277425    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb41e2-890d-4add-86b5-39d8581198d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.278475 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.278425    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.278597 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.278485    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb41e2-890d-4add-86b5-39d8581198d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.278855 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.278831    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.279086 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.279064    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.279390 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.279366    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.282750 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.282700    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.283031 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.283009    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.303048 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.303016    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.303737 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.303709    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z762t\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-kube-api-access-z762t\") pod \"maas-default-gateway-openshift-default-6886b96f9-hdb7l\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.378453 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.378383    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a57390c7-mkggv\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.378605 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.378580    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a57390c7-mkggv\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.378668 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.378635    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646fp\" (UniqueName: \"kubernetes.io/projected/a9c92f72-a156-4100-b7e7-28a41e871bb5-kube-api-access-646fp\") pod \"maas-api-revoke-sub-a57390c7-mkggv\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.404912 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.404885    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:16.485809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.485749    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a57390c7-mkggv\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.485973 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.485829    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646fp\" (UniqueName: \"kubernetes.io/projected/a9c92f72-a156-4100-b7e7-28a41e871bb5-kube-api-access-646fp\") pod \"maas-api-revoke-sub-a57390c7-mkggv\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.485973 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.485954    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a57390c7-mkggv\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.487582 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.487554    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a57390c7-mkggv\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.489644 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.489620    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a57390c7-mkggv\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.512231 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.512199    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-646fp\" (UniqueName: \"kubernetes.io/projected/a9c92f72-a156-4100-b7e7-28a41e871bb5-kube-api-access-646fp\") pod \"maas-api-revoke-sub-a57390c7-mkggv\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.549590 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.549567    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:16.734625 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:57:16.734584    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cbb41e2_890d_4add_86b5_39d8581198d2.slice/crio-02ff1f81f036d53c07e20f8ad59a4530e5106010257ba1ecd792a5574ca83945 WatchSource:0}: Error finding container 02ff1f81f036d53c07e20f8ad59a4530e5106010257ba1ecd792a5574ca83945: Status 404 returned error can't find the container with id 02ff1f81f036d53c07e20f8ad59a4530e5106010257ba1ecd792a5574ca83945
Nov 02 13:57:16.734757 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.734636    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"]
Nov 02 13:57:16.737033 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.737001    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:57:16.737150 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.737075    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:57:16.737150 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.737119    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:57:16.862091 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:57:16.862061    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c92f72_a156_4100_b7e7_28a41e871bb5.slice/crio-d6fff0bd2d86722bc92e229642652dc59538987c7ef99eab8730d574c6b168d3 WatchSource:0}: Error finding container d6fff0bd2d86722bc92e229642652dc59538987c7ef99eab8730d574c6b168d3: Status 404 returned error can't find the container with id d6fff0bd2d86722bc92e229642652dc59538987c7ef99eab8730d574c6b168d3
Nov 02 13:57:16.864123 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:16.863765    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"]
Nov 02 13:57:17.523927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:17.523894    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv" event={"ID":"a9c92f72-a156-4100-b7e7-28a41e871bb5","Type":"ContainerStarted","Data":"941a3e7a9aeceb8fdae9f2ed8b863746e90d8cdef100d8c921424ca59db17970"}
Nov 02 13:57:17.524429 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:17.523945    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv" event={"ID":"a9c92f72-a156-4100-b7e7-28a41e871bb5","Type":"ContainerStarted","Data":"d6fff0bd2d86722bc92e229642652dc59538987c7ef99eab8730d574c6b168d3"}
Nov 02 13:57:17.525620 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:17.525592    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" event={"ID":"7cbb41e2-890d-4add-86b5-39d8581198d2","Type":"ContainerStarted","Data":"a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1"}
Nov 02 13:57:17.525733 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:17.525643    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" event={"ID":"7cbb41e2-890d-4add-86b5-39d8581198d2","Type":"ContainerStarted","Data":"02ff1f81f036d53c07e20f8ad59a4530e5106010257ba1ecd792a5574ca83945"}
Nov 02 13:57:17.565480 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:17.565358    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv" podStartSLOduration=1.5653452030000001 podStartE2EDuration="1.565345203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:17.564719291 +0000 UTC m=+828.316023729" watchObservedRunningTime="2026-11-02 13:57:17.565345203 +0000 UTC m=+828.316649692"
Nov 02 13:57:17.621572 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:17.621512    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" podStartSLOduration=1.6214971139999999 podStartE2EDuration="1.621497114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:17.619942514 +0000 UTC m=+828.371246951" watchObservedRunningTime="2026-11-02 13:57:17.621497114 +0000 UTC m=+828.372801551"
Nov 02 13:57:18.412908 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:18.410076    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:18.416982 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:18.414871    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:18.416982 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:18.414934    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 13:57:18.443148 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:18.443096    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:57:18.534694 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:18.534660    5034 generic.go:356] "Generic (PLEG): container finished" podID="a9c92f72-a156-4100-b7e7-28a41e871bb5" containerID="941a3e7a9aeceb8fdae9f2ed8b863746e90d8cdef100d8c921424ca59db17970" exitCode=0
Nov 02 13:57:18.535115 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:18.534741    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv" event={"ID":"a9c92f72-a156-4100-b7e7-28a41e871bb5","Type":"ContainerDied","Data":"941a3e7a9aeceb8fdae9f2ed8b863746e90d8cdef100d8c921424ca59db17970"}
Nov 02 13:57:19.416361 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.416325    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:19.416522 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.416387    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 13:57:19.810448 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.810419    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:19.850032 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.850002    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-xks-ca\") pod \"a9c92f72-a156-4100-b7e7-28a41e871bb5\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") "
Nov 02 13:57:19.850184 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.850062    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c92f72-a156-4100-b7e7-28a41e871bb5-kube-api-access-646fp\" (UniqueName: \"kubernetes.io/projected/a9c92f72-a156-4100-b7e7-28a41e871bb5-kube-api-access-646fp\") pod \"a9c92f72-a156-4100-b7e7-28a41e871bb5\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") "
Nov 02 13:57:19.850238 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.850228    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-openshift-ca\") pod \"a9c92f72-a156-4100-b7e7-28a41e871bb5\" (UID: \"a9c92f72-a156-4100-b7e7-28a41e871bb5\") "
Nov 02 13:57:19.850610 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.850574    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-openshift-ca" pod "a9c92f72-a156-4100-b7e7-28a41e871bb5" (UID: "a9c92f72-a156-4100-b7e7-28a41e871bb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:19.854526 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.854497    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c92f72-a156-4100-b7e7-28a41e871bb5-kube-api-access-646fp" pod "a9c92f72-a156-4100-b7e7-28a41e871bb5" (UID: "a9c92f72-a156-4100-b7e7-28a41e871bb5"). InnerVolumeSpecName "kube-api-access-646fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:19.856385 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.856361    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-xks-ca" pod "a9c92f72-a156-4100-b7e7-28a41e871bb5" (UID: "a9c92f72-a156-4100-b7e7-28a41e871bb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:19.951402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.951371    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:19.951402 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.951401    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646fp\" (UniqueName: \"kubernetes.io/projected/a9c92f72-a156-4100-b7e7-28a41e871bb5-kube-api-access-646fp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:19.951591 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:19.951415    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c92f72-a156-4100-b7e7-28a41e871bb5-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:20.407968 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:20.407885    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:20.407968 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:20.407932    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 13:57:20.553151 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:20.553118    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"
Nov 02 13:57:20.553316 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:20.553113    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv" event={"ID":"a9c92f72-a156-4100-b7e7-28a41e871bb5","Type":"ContainerDied","Data":"d6fff0bd2d86722bc92e229642652dc59538987c7ef99eab8730d574c6b168d3"}
Nov 02 13:57:20.553316 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:20.553237    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6fff0bd2d86722bc92e229642652dc59538987c7ef99eab8730d574c6b168d3"
Nov 02 13:57:21.405859 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:21.405831    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:21.406241 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:21.405883    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 13:57:22.405990 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:22.405959    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:22.406532 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:22.406018    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 13:57:23.405511 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:23.405476    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hdb7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 02 13:57:23.405689 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:23.405538    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 02 13:57:25.288596 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:25.288559    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:25.289269 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:25.289246    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:25.290601 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:25.290553    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 13:57:38.538303 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.538268    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"]
Nov 02 13:57:38.544826 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.539766    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a9c92f72-a156-4100-b7e7-28a41e871bb5" containerName="revoke-keys"
Nov 02 13:57:38.545008 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.544988    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c92f72-a156-4100-b7e7-28a41e871bb5" containerName="revoke-keys"
Nov 02 13:57:38.545328 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.545309    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c92f72-a156-4100-b7e7-28a41e871bb5" containerName="revoke-keys"
Nov 02 13:57:38.554860 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.554838    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:38.560134 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.560104    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"]
Nov 02 13:57:38.733809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.728067    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:57:38.733809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.728229    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3eee531c3de6\""
Nov 02 13:57:38.733809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.728290    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-892a32dda57a\""
Nov 02 13:57:38.733809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.728471    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lzlpd\""
Nov 02 13:57:38.733809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.728526    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:57:38.733809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.728842    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:57:38.758801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.754169    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-plugins-config-volume\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:38.758801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.754248    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:57:38.869867 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.867188    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:38.869867 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.867268    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-routing-overlay-volume\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:38.869867 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:38.867321    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-449bm\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-kube-api-access-449bm\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.032801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.018575    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.032801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.023071    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-routing-overlay-volume\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.032801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.023888    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-routing-overlay-volume\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.040803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.040629    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-449bm\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-kube-api-access-449bm\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.040803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.040756    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-plugins-config-volume\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.045422 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.041703    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-plugins-config-volume\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.045422 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.043880    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.074022 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.073702    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-449bm\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-kube-api-access-449bm\") pod \"payload-processing-external-model-86648b6d67-sr7vr\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") " pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.248763 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.248395    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.614515 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.614480    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"]
Nov 02 13:57:39.923465 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.923377    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr" event={"ID":"2cf411dd-fe9d-4518-88c5-9ad22fa2be45","Type":"ContainerStarted","Data":"d023c8bf9d090486f88203482b0bd84ca9665f587135bb041427e2b7257aa840"}
Nov 02 13:57:39.923609 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.923471    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:39.923609 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.923495    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr" event={"ID":"2cf411dd-fe9d-4518-88c5-9ad22fa2be45","Type":"ContainerStarted","Data":"9e6686c821fe9710032ce7d340637d47224120de0c13846e6f8770477f4882b3"}
Nov 02 13:57:39.967769 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:39.967710    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr" podStartSLOduration=1.967695767 podStartE2EDuration="1.967695767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:39.966531422 +0000 UTC m=+850.717835863" watchObservedRunningTime="2026-11-02 13:57:39.967695767 +0000 UTC m=+850.719000205"
Nov 02 13:57:40.740767 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:40.740675    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"]
Nov 02 13:57:40.986179 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:40.984929    5034 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-892a32dda57a: secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:40.986179 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:40.984958    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr: secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:40.986179 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:40.985065    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials podName:2cf411dd-fe9d-4518-88c5-9ad22fa2be45 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:41.485046529 +0000 UTC m=+852.236350961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials") pod "payload-processing-external-model-86648b6d67-sr7vr" (UID: "2cf411dd-fe9d-4518-88c5-9ad22fa2be45") : secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:41.490046 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:41.490017    5034 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-892a32dda57a: secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:41.490046 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:41.490039    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr: secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:41.490242 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:41.490097    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials podName:2cf411dd-fe9d-4518-88c5-9ad22fa2be45 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:42.490081965 +0000 UTC m=+853.241386380 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials") pod "payload-processing-external-model-86648b6d67-sr7vr" (UID: "2cf411dd-fe9d-4518-88c5-9ad22fa2be45") : secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:42.337386 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.337347    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"]
Nov 02 13:57:42.341690 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.341672    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.377577 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.377545    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"]
Nov 02 13:57:42.402374 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.402344    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.402529 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.402389    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.402529 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.402477    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.402656 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.402536    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.402656 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.402603    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.402737 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.402672    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.402737 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.402722    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7wtt\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-kube-api-access-z7wtt\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.402860 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.402813    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.402916 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.402906    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.504447 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.504413    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.504603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.504467    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.504603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.504509    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.504603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.504556    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.504603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.504584    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.504834 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.504632    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.504834 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.504658    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7wtt\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-kube-api-access-z7wtt\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.504834 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.504730    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.504993 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.504860    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.505051 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.505032    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.505169 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:42.505143    5034 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-892a32dda57a: secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:42.505169 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:42.505165    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr: secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:42.505301 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.505217    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.505301 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:42.505231    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials podName:2cf411dd-fe9d-4518-88c5-9ad22fa2be45 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:44.505213185 +0000 UTC m=+855.256517607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials") pod "payload-processing-external-model-86648b6d67-sr7vr" (UID: "2cf411dd-fe9d-4518-88c5-9ad22fa2be45") : secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:42.505400 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.505299    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.505400 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.505346    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.505513 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.505475    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.507561 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.507530    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.507699 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.507680    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.531634 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.531610    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7wtt\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-kube-api-access-z7wtt\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.533530 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.533510    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.656330 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.656251    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:42.981679 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:57:42.981650    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4226b2d7_e1e2_4e5f_bcc6_7401535821b5.slice/crio-9ad11540e0e8cfd65153f3421f17e49d0903c08bd6217eb49dd7fcda32193670 WatchSource:0}: Error finding container 9ad11540e0e8cfd65153f3421f17e49d0903c08bd6217eb49dd7fcda32193670: Status 404 returned error can't find the container with id 9ad11540e0e8cfd65153f3421f17e49d0903c08bd6217eb49dd7fcda32193670
Nov 02 13:57:42.982420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.982397    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"]
Nov 02 13:57:42.984391 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.984360    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:57:42.984495 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.984420    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:57:42.984495 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:42.984450    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:57:43.949628 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:43.949595    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" event={"ID":"4226b2d7-e1e2-4e5f-bcc6-7401535821b5","Type":"ContainerStarted","Data":"25eb5e59090bd31eb2022f1ef97a6c6b1409b81f338cc52cdb89e293cefb184f"}
Nov 02 13:57:43.950045 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:43.949654    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" event={"ID":"4226b2d7-e1e2-4e5f-bcc6-7401535821b5","Type":"ContainerStarted","Data":"9ad11540e0e8cfd65153f3421f17e49d0903c08bd6217eb49dd7fcda32193670"}
Nov 02 13:57:44.026937 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:44.026882    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podStartSLOduration=2.026869044 podStartE2EDuration="2.026869044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:44.012964528 +0000 UTC m=+854.764268965" watchObservedRunningTime="2026-11-02 13:57:44.026869044 +0000 UTC m=+854.778173482"
Nov 02 13:57:44.529704 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:44.529671    5034 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-892a32dda57a: secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:44.529704 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:44.529698    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr: secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:44.529948 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:44.529753    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials podName:2cf411dd-fe9d-4518-88c5-9ad22fa2be45 nodeName:}" failed. No retries permitted until 2026-11-02 13:57:48.529737258 +0000 UTC m=+859.281041677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials") pod "payload-processing-external-model-86648b6d67-sr7vr" (UID: "2cf411dd-fe9d-4518-88c5-9ad22fa2be45") : secret "agc-cred-892a32dda57a" not found
Nov 02 13:57:44.657075 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:44.657043    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:44.658262 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:44.658232    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:44.658489 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:44.658440    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 13:57:45.669801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:45.665162    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:45.669801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:45.665233    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 13:57:46.657116 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:46.657083    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:46.657354 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:46.657332    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 13:57:46.936949 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:46.936848    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr" podUID="2cf411dd-fe9d-4518-88c5-9ad22fa2be45" containerName="payload-processing" containerID="cri-o://d023c8bf9d090486f88203482b0bd84ca9665f587135bb041427e2b7257aa840" gracePeriod=25
Nov 02 13:57:47.694832 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:47.686902    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:47.694832 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:47.686973    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 13:57:48.127732 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.121363    5034 generic.go:356] "Generic (PLEG): container finished" podID="2cf411dd-fe9d-4518-88c5-9ad22fa2be45" containerID="d023c8bf9d090486f88203482b0bd84ca9665f587135bb041427e2b7257aa840" exitCode=0
Nov 02 13:57:48.127732 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.121530    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr" event={"ID":"2cf411dd-fe9d-4518-88c5-9ad22fa2be45","Type":"ContainerDied","Data":"d023c8bf9d090486f88203482b0bd84ca9665f587135bb041427e2b7257aa840"}
Nov 02 13:57:48.188987 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.188956    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:48.297287 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.297251    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-plugins-config-volume\") pod \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") "
Nov 02 13:57:48.297609 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.297585    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-kube-api-access-449bm\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-kube-api-access-449bm\") pod \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") "
Nov 02 13:57:48.298007 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.297984    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-routing-overlay-volume\") pod \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") "
Nov 02 13:57:48.298178 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.298160    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials\") pod \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\" (UID: \"2cf411dd-fe9d-4518-88c5-9ad22fa2be45\") "
Nov 02 13:57:48.300469 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.300442    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-plugins-config-volume" pod "2cf411dd-fe9d-4518-88c5-9ad22fa2be45" (UID: "2cf411dd-fe9d-4518-88c5-9ad22fa2be45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:48.300754 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.300733    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-routing-overlay-volume" pod "2cf411dd-fe9d-4518-88c5-9ad22fa2be45" (UID: "2cf411dd-fe9d-4518-88c5-9ad22fa2be45"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:48.318572 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.318532    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-kube-api-access-449bm" pod "2cf411dd-fe9d-4518-88c5-9ad22fa2be45" (UID: "2cf411dd-fe9d-4518-88c5-9ad22fa2be45"). InnerVolumeSpecName "kube-api-access-449bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:48.340759 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.336966    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials" pod "2cf411dd-fe9d-4518-88c5-9ad22fa2be45" (UID: "2cf411dd-fe9d-4518-88c5-9ad22fa2be45"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:48.443804 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.438558    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-449bm\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-kube-api-access-449bm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:48.443804 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.438601    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:48.443804 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.438620    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:48.443804 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.438635    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf411dd-fe9d-4518-88c5-9ad22fa2be45-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:57:48.503062 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.502920    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:48.668861 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.667055    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:48.668861 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.667209    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 13:57:48.668861 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.667760    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 13:57:48.688934 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.688863    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:48.807837 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:57:48.786699    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:57:48.808035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.807942    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:48.892981 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:48.892925    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:49.120880 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.120836    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:49.516190 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.516154    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"
Nov 02 13:57:49.547903 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.542719    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:49.588753 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.588707    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr" event={"ID":"2cf411dd-fe9d-4518-88c5-9ad22fa2be45","Type":"ContainerDied","Data":"9e6686c821fe9710032ce7d340637d47224120de0c13846e6f8770477f4882b3"}
Nov 02 13:57:49.589095 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.589072    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:57:49.591165 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.590325    5034 scope.go:122] "RemoveContainer" containerID="d023c8bf9d090486f88203482b0bd84ca9665f587135bb041427e2b7257aa840"
Nov 02 13:57:49.662051 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.662019    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:49.662295 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.662268    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 13:57:49.690906 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.690860    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:49.691425 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.691405    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:57:49.718802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.716873    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:49.718802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.717268    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:57:49.827854 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.827715    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:49.835797 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.828472    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:57:49.845634 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.845600    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:57:49.907380 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:49.906750    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:57:50.035750 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:50.033055    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"]
Nov 02 13:57:50.046453 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:50.046365    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-86648b6d67-sr7vr"]
Nov 02 13:57:50.191942 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:50.169053    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:57:50.191942 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:50.190859    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 13:57:50.191942 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:50.191397    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:57:50.217000 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:50.214089    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:57:50.231454 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:50.231433    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 13:57:50.656608 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:50.656576    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 02 13:57:50.657039 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:50.656638    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 02 13:57:51.395068 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:51.395038    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf411dd-fe9d-4518-88c5-9ad22fa2be45" path="/var/lib/kubelet/pods/2cf411dd-fe9d-4518-88c5-9ad22fa2be45/volumes"
Nov 02 13:57:51.662476 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:51.662402    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:51.662972 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:51.662871    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:57:51.663748 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:57:51.663724    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 13:58:09.818613 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.818577    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"]
Nov 02 13:58:09.819490 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.819468    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2cf411dd-fe9d-4518-88c5-9ad22fa2be45" containerName="payload-processing"
Nov 02 13:58:09.819490 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.819491    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf411dd-fe9d-4518-88c5-9ad22fa2be45" containerName="payload-processing"
Nov 02 13:58:09.819969 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.819665    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf411dd-fe9d-4518-88c5-9ad22fa2be45" containerName="payload-processing"
Nov 02 13:58:09.824292 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.824272    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:09.833415 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.833365    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"]
Nov 02 13:58:09.901860 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.901829    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8hv\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-kube-api-access-hl8hv\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:09.902035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.901905    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:09.902035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.902012    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:09.902159 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.902069    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:09.902220 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.902163    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:09.902220 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.902208    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:09.902306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.902282    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:09.902394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.902379    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:09.902450 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:09.902436    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.003273 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003242    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.003455 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003322    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8hv\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-kube-api-access-hl8hv\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.003455 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003409    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.003455 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003446    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.003629 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003486    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.003629 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003518    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.003629 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003549    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.003629 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003575    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.003902 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003649    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.004085 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.003997    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.004822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.004174    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.004822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.004331    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.004822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.004461    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.004822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.004758    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.005885 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.005859    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.006554 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.006535    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.028046 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.028022    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8hv\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-kube-api-access-hl8hv\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.028322 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.028303    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.142121 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.142042    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:10.469550 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:58:10.469507    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fdf194_1d79_4618_a09b_2fc9c16d94eb.slice/crio-0aee75176f912173e739ff7b40a84989ff3d7666afdcfc14f95a4dc340157947 WatchSource:0}: Error finding container 0aee75176f912173e739ff7b40a84989ff3d7666afdcfc14f95a4dc340157947: Status 404 returned error can't find the container with id 0aee75176f912173e739ff7b40a84989ff3d7666afdcfc14f95a4dc340157947
Nov 02 13:58:10.470598 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.470570    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"]
Nov 02 13:58:10.472444 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.472408    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:58:10.472526 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.472492    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:58:10.472577 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.472560    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:58:10.728864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.722984    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" event={"ID":"48fdf194-1d79-4618-a09b-2fc9c16d94eb","Type":"ContainerStarted","Data":"fedc9bd63df4d6140957b55ffc34a61bdb36e023d76138716a98aa22a626d7a5"}
Nov 02 13:58:10.728864 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.723090    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" event={"ID":"48fdf194-1d79-4618-a09b-2fc9c16d94eb","Type":"ContainerStarted","Data":"0aee75176f912173e739ff7b40a84989ff3d7666afdcfc14f95a4dc340157947"}
Nov 02 13:58:10.776824 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:10.776632    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podStartSLOduration=1.776612826 podStartE2EDuration="1.776612826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:10.775319936 +0000 UTC m=+881.526624373" watchObservedRunningTime="2026-11-02 13:58:10.776612826 +0000 UTC m=+881.527917265"
Nov 02 13:58:11.142538 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:11.142504    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:11.144088 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:11.144063    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:11.144202 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:11.144114    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 13:58:12.144053 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:12.143656    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:12.144053 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:12.143716    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 13:58:13.157801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:13.157088    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:13.157801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:13.157145    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 13:58:14.143146 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:14.143113    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:14.143322 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:14.143168    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 13:58:15.142730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:15.142697    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:15.143146 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:15.142748    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 13:58:16.143285 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:16.143248    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:16.143806 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:16.143299    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 13:58:17.142952 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:17.142922    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 02 13:58:17.143144 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:17.142977    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 02 13:58:18.147190 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.147162    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:18.147633 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.147621    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:18.148246 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.148225    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 13:58:18.414559 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:58:18.414472    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:58:18.488521 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.488486    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"]
Nov 02 13:58:18.492809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.492773    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.508409 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.508364    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:58:18.508409 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.508398    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:58:18.508676 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.508426    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:18.508676 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.508530    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:58:18.512892 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.512858    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"]
Nov 02 13:58:18.595267 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.595233    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdxg\" (UniqueName: \"kubernetes.io/projected/9ec00317-2f7d-49b7-b976-1bdd135c6352-kube-api-access-hrdxg\") pod \"maas-api-revoke-sub-baa44ce7-5wk9d\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.595435 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.595376    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa44ce7-5wk9d\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.595488 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.595461    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa44ce7-5wk9d\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.696588 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.696504    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa44ce7-5wk9d\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.696736 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.696701    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdxg\" (UniqueName: \"kubernetes.io/projected/9ec00317-2f7d-49b7-b976-1bdd135c6352-kube-api-access-hrdxg\") pod \"maas-api-revoke-sub-baa44ce7-5wk9d\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.696827 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.696761    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa44ce7-5wk9d\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.697202 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.697178    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa44ce7-5wk9d\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.699049 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.699027    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa44ce7-5wk9d\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.720519 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.720494    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdxg\" (UniqueName: \"kubernetes.io/projected/9ec00317-2f7d-49b7-b976-1bdd135c6352-kube-api-access-hrdxg\") pod \"maas-api-revoke-sub-baa44ce7-5wk9d\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:18.813744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:18.813713    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:19.123259 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:58:19.123208    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec00317_2f7d_49b7_b976_1bdd135c6352.slice/crio-58775be20e0b5a72301ed78d524bf548703181c76340ef1b6fab05267f6faebb WatchSource:0}: Error finding container 58775be20e0b5a72301ed78d524bf548703181c76340ef1b6fab05267f6faebb: Status 404 returned error can't find the container with id 58775be20e0b5a72301ed78d524bf548703181c76340ef1b6fab05267f6faebb
Nov 02 13:58:19.124671 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:19.124606    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"]
Nov 02 13:58:19.803877 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:19.803842    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d" event={"ID":"9ec00317-2f7d-49b7-b976-1bdd135c6352","Type":"ContainerStarted","Data":"b78230db0456b0d3719cf3d8976a58d73838302f2caa68ba6026639d4bbd34b9"}
Nov 02 13:58:19.804360 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:19.803895    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d" event={"ID":"9ec00317-2f7d-49b7-b976-1bdd135c6352","Type":"ContainerStarted","Data":"58775be20e0b5a72301ed78d524bf548703181c76340ef1b6fab05267f6faebb"}
Nov 02 13:58:19.846329 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:19.846285    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d" podStartSLOduration=1.846271435 podStartE2EDuration="1.846271435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:19.844298762 +0000 UTC m=+890.595603191" watchObservedRunningTime="2026-11-02 13:58:19.846271435 +0000 UTC m=+890.597575873"
Nov 02 13:58:20.847749 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:20.847709    5034 generic.go:356] "Generic (PLEG): container finished" podID="9ec00317-2f7d-49b7-b976-1bdd135c6352" containerID="b78230db0456b0d3719cf3d8976a58d73838302f2caa68ba6026639d4bbd34b9" exitCode=0
Nov 02 13:58:20.848375 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:20.847903    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d" event={"ID":"9ec00317-2f7d-49b7-b976-1bdd135c6352","Type":"ContainerDied","Data":"b78230db0456b0d3719cf3d8976a58d73838302f2caa68ba6026639d4bbd34b9"}
Nov 02 13:58:22.029028 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.028998    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:22.153677 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.153596    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-openshift-ca\") pod \"9ec00317-2f7d-49b7-b976-1bdd135c6352\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") "
Nov 02 13:58:22.153677 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.153664    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-xks-ca\") pod \"9ec00317-2f7d-49b7-b976-1bdd135c6352\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") "
Nov 02 13:58:22.153912 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.153730    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec00317-2f7d-49b7-b976-1bdd135c6352-kube-api-access-hrdxg\" (UniqueName: \"kubernetes.io/projected/9ec00317-2f7d-49b7-b976-1bdd135c6352-kube-api-access-hrdxg\") pod \"9ec00317-2f7d-49b7-b976-1bdd135c6352\" (UID: \"9ec00317-2f7d-49b7-b976-1bdd135c6352\") "
Nov 02 13:58:22.153982 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.153966    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-openshift-ca" pod "9ec00317-2f7d-49b7-b976-1bdd135c6352" (UID: "9ec00317-2f7d-49b7-b976-1bdd135c6352"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:22.154182 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.154165    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:58:22.155848 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.155825    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-xks-ca" pod "9ec00317-2f7d-49b7-b976-1bdd135c6352" (UID: "9ec00317-2f7d-49b7-b976-1bdd135c6352"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:22.155942 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.155831    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec00317-2f7d-49b7-b976-1bdd135c6352-kube-api-access-hrdxg" pod "9ec00317-2f7d-49b7-b976-1bdd135c6352" (UID: "9ec00317-2f7d-49b7-b976-1bdd135c6352"). InnerVolumeSpecName "kube-api-access-hrdxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:22.255393 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.255365    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ec00317-2f7d-49b7-b976-1bdd135c6352-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:58:22.255393 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.255392    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrdxg\" (UniqueName: \"kubernetes.io/projected/9ec00317-2f7d-49b7-b976-1bdd135c6352-kube-api-access-hrdxg\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:58:22.863187 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.863151    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"
Nov 02 13:58:22.863187 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.863168    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d" event={"ID":"9ec00317-2f7d-49b7-b976-1bdd135c6352","Type":"ContainerDied","Data":"58775be20e0b5a72301ed78d524bf548703181c76340ef1b6fab05267f6faebb"}
Nov 02 13:58:22.863456 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:22.863209    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58775be20e0b5a72301ed78d524bf548703181c76340ef1b6fab05267f6faebb"
Nov 02 13:58:29.485319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:29.485290    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 13:58:29.490097 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:29.490068    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 13:58:31.352079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.352046    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"]
Nov 02 13:58:31.352613 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.352597    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9ec00317-2f7d-49b7-b976-1bdd135c6352" containerName="revoke-keys"
Nov 02 13:58:31.352692 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.352614    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec00317-2f7d-49b7-b976-1bdd135c6352" containerName="revoke-keys"
Nov 02 13:58:31.352763 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.352751    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec00317-2f7d-49b7-b976-1bdd135c6352" containerName="revoke-keys"
Nov 02 13:58:31.365302 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.365279    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.379246 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.379221    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"]
Nov 02 13:58:31.394681 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.394652    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"distinct-2-w3-f30173-kserve-self-signed-certs\""
Nov 02 13:58:31.394681 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.394657    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"openshift-service-ca.crt\""
Nov 02 13:58:31.396316 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.396294    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"kube-root-ca.crt\""
Nov 02 13:58:31.461752 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.461727    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tls-certs\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.461894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.461760    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-model-cache\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.461894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.461847    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7ltd\" (UniqueName: \"kubernetes.io/projected/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-kube-api-access-z7ltd\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.461894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.461882    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tmp-dir\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.462005 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.461995    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-dshm\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.462040 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.462024    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-home\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.482650 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.482628    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f30173\"/\"default-dockercfg-s5wk6\""
Nov 02 13:58:31.563387 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.563346    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-dshm\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.564509 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.564477    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-home\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.564742 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.564721    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tls-certs\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.564839 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.564761    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-model-cache\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.566078 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.566054    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-home\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.566230 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.566106    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7ltd\" (UniqueName: \"kubernetes.io/projected/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-kube-api-access-z7ltd\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.568093 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.566466    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tmp-dir\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.568093 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.567267    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tmp-dir\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.568315 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.568291    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-model-cache\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.590801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.573444    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-dshm\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.590801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.574042    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tls-certs\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.616848 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.614723    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7ltd\" (UniqueName: \"kubernetes.io/projected/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-kube-api-access-z7ltd\") pod \"distinct-2-w3-f30173-kserve-7679bb87-cpjcg\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") " pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.681819 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.681785    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:31.996203 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:31.995886    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"]
Nov 02 13:58:31.996910 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:58:31.996878    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20351fa4_e1a5_4e28_b2fe_750b5fbf8a68.slice/crio-73b02223f73acdf6f086824ff2cc677dd02593a42f4c45740135bd5e9aadcaf5 WatchSource:0}: Error finding container 73b02223f73acdf6f086824ff2cc677dd02593a42f4c45740135bd5e9aadcaf5: Status 404 returned error can't find the container with id 73b02223f73acdf6f086824ff2cc677dd02593a42f4c45740135bd5e9aadcaf5
Nov 02 13:58:32.950816 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:32.949409    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" event={"ID":"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68","Type":"ContainerStarted","Data":"6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b"}
Nov 02 13:58:32.950816 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:32.949458    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" event={"ID":"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68","Type":"ContainerStarted","Data":"73b02223f73acdf6f086824ff2cc677dd02593a42f4c45740135bd5e9aadcaf5"}
Nov 02 13:58:33.682878 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:33.682842    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:41.684804 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:41.684224    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:41.947858 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:41.945084    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:41.998800 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:41.997952    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podStartSLOduration=10.997932382 podStartE2EDuration="10.997932382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:33.000146419 +0000 UTC m=+903.751450884" watchObservedRunningTime="2026-11-02 13:58:41.997932382 +0000 UTC m=+912.749236822"
Nov 02 13:58:42.140803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:42.140084    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:58:42.902803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:42.901877    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:44.271802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:44.270919    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:44.734474 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:44.733928    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9f9xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:44.734474 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:44.734010    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:47.761188 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.761156    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"]
Nov 02 13:58:47.765398 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.765380    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.810764 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.810732    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-92f249\"/\"distinct-2-w1-92f249-kserve-self-signed-certs\""
Nov 02 13:58:47.814186 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.814159    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"]
Nov 02 13:58:47.837760 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.837734    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-dshm\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.837885 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.837863    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-home\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.837948 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.837909    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmc2x\" (UniqueName: \"kubernetes.io/projected/08a21e6a-28af-4b2e-9251-32f455208189-kube-api-access-rmc2x\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.838035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.838016    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-tmp-dir\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.838100 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.838088    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a21e6a-28af-4b2e-9251-32f455208189-tls-certs\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.838253 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.838225    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-model-cache\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.945957 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.941506    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-tmp-dir\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.945957 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.941600    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a21e6a-28af-4b2e-9251-32f455208189-tls-certs\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.945957 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.941688    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-model-cache\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.945957 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.941805    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-dshm\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.945957 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.941878    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-home\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.945957 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.941907    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmc2x\" (UniqueName: \"kubernetes.io/projected/08a21e6a-28af-4b2e-9251-32f455208189-kube-api-access-rmc2x\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.945957 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.942653    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-tmp-dir\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.959330 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.959297    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-model-cache\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.959871 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.959847    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-home\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.974492 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.974459    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-dshm\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.975507 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.975482    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmc2x\" (UniqueName: \"kubernetes.io/projected/08a21e6a-28af-4b2e-9251-32f455208189-kube-api-access-rmc2x\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:47.978030 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:47.978004    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a21e6a-28af-4b2e-9251-32f455208189-tls-certs\") pod \"distinct-2-w1-92f249-kserve-bcc58ff59-dc4md\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") " pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:48.078771 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:48.078688    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:48.389927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:48.389883    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"]
Nov 02 13:58:48.401041 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:58:48.400998    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08a21e6a_28af_4b2e_9251_32f455208189.slice/crio-76e76df2e7c2812b5023dc40b9d748df64b19870b9d003f8c3eb1550f0b39ee3 WatchSource:0}: Error finding container 76e76df2e7c2812b5023dc40b9d748df64b19870b9d003f8c3eb1550f0b39ee3: Status 404 returned error can't find the container with id 76e76df2e7c2812b5023dc40b9d748df64b19870b9d003f8c3eb1550f0b39ee3
Nov 02 13:58:48.425825 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:58:48.425757    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:58:49.142357 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:49.142321    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" event={"ID":"08a21e6a-28af-4b2e-9251-32f455208189","Type":"ContainerStarted","Data":"c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588"}
Nov 02 13:58:49.142841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:49.142371    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" event={"ID":"08a21e6a-28af-4b2e-9251-32f455208189","Type":"ContainerStarted","Data":"76e76df2e7c2812b5023dc40b9d748df64b19870b9d003f8c3eb1550f0b39ee3"}
Nov 02 13:58:49.207846 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:49.207772    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podStartSLOduration=2.207759462 podStartE2EDuration="2.207759462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:49.205421691 +0000 UTC m=+919.956726128" watchObservedRunningTime="2026-11-02 13:58:49.207759462 +0000 UTC m=+919.959063936"
Nov 02 13:58:50.080312 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:50.080274    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:53.174803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:53.173863    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:58:55.002459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.002425    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"]
Nov 02 13:58:55.007071 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.007049    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.044047 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.044018    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"]
Nov 02 13:58:55.084521 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.084491    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.084691 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.084532    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.084691 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.084657    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.084798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.084712    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.084798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.084731    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.084875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.084852    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcchz\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-kube-api-access-gcchz\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.084914 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.084891    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.084948 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.084939    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.085002 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.084990    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186103 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186070    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186103 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186108    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186350 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186145    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcchz\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-kube-api-access-gcchz\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186350 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186174    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186350 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186202    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186350 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186254    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186350 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186322    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186615 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186427    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186615 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186536    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186717 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186645    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.186985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186961    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.187079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186987    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.187079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186991    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.187079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.186996    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.188592 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.188575    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.188877 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.188849    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.211121 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.211096    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcchz\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-kube-api-access-gcchz\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.211517 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.211478    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.328423 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.328349    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:55.660408 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:58:55.660377    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d97ee28_1ccf_4ce4_8c26_5c50a0ccf4ec.slice/crio-d1636d02b2398680600dbb1669419236795254907f14e0ddd5500b49702a3d57 WatchSource:0}: Error finding container d1636d02b2398680600dbb1669419236795254907f14e0ddd5500b49702a3d57: Status 404 returned error can't find the container with id d1636d02b2398680600dbb1669419236795254907f14e0ddd5500b49702a3d57
Nov 02 13:58:55.663090 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.663057    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:58:55.663182 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.663128    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:58:55.663182 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.663155    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:58:55.663270 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:55.663253    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"]
Nov 02 13:58:56.415178 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:56.415139    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" event={"ID":"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec","Type":"ContainerStarted","Data":"4c9813146911ff714bfffb146239169043c41271742a8ada3e607337d544b99a"}
Nov 02 13:58:56.415568 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:56.415203    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" event={"ID":"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec","Type":"ContainerStarted","Data":"d1636d02b2398680600dbb1669419236795254907f14e0ddd5500b49702a3d57"}
Nov 02 13:58:56.494594 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:56.494515    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podStartSLOduration=2.49449416 podStartE2EDuration="2.49449416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:58:56.480407038 +0000 UTC m=+927.231711476" watchObservedRunningTime="2026-11-02 13:58:56.49449416 +0000 UTC m=+927.245798601"
Nov 02 13:58:57.328907 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:57.328877    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:58:57.330663 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:57.330631    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 02 13:58:57.330829 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:57.330687    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 02 13:58:58.089329 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:58.088093    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:58.127617 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:58.127583    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:58.328744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:58.328711    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 02 13:58:58.328942 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:58.328761    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 02 13:58:58.446617 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:58.446543    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 13:58:59.329696 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:59.329659    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 02 13:58:59.330131 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:58:59.329727    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 02 13:59:00.329625 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:00.329593    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 02 13:59:00.329845 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:00.329642    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 02 13:59:01.329437 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.329396    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 02 13:59:01.329622 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.329453    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 02 13:59:01.757311 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.757270    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"]
Nov 02 13:59:01.761909 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.761888    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:01.793284 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.793255    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"]
Nov 02 13:59:01.796591 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.796426    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:59:01.796591 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.796530    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:59:01.796591 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.796562    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:01.796981 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.796950    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:59:01.880237 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.880205    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnq8w\" (UniqueName: \"kubernetes.io/projected/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-kube-api-access-pnq8w\") pod \"maas-api-revoke-sub-8ba4ccb4-d82lx\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:01.880437 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.880347    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba4ccb4-d82lx\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:01.880437 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.880415    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba4ccb4-d82lx\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:01.981166 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.981129    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba4ccb4-d82lx\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:01.981361 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.981179    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba4ccb4-d82lx\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:01.981361 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.981319    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnq8w\" (UniqueName: \"kubernetes.io/projected/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-kube-api-access-pnq8w\") pod \"maas-api-revoke-sub-8ba4ccb4-d82lx\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:01.982060 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.982034    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ba4ccb4-d82lx\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:01.983791 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:01.983755    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ba4ccb4-d82lx\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:02.009411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:02.009345    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnq8w\" (UniqueName: \"kubernetes.io/projected/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-kube-api-access-pnq8w\") pod \"maas-api-revoke-sub-8ba4ccb4-d82lx\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:02.078671 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:02.078634    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:02.329079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:02.328981    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 02 13:59:02.329079 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:02.329062    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 02 13:59:02.447177 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:02.447142    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"]
Nov 02 13:59:02.534491 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:02.534450    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx" event={"ID":"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4","Type":"ContainerStarted","Data":"67a44842c4840aacd6cc1a74df7466c06a6b3198c8c4505b393f966a4a01c7e4"}
Nov 02 13:59:03.335118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:03.335085    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:59:03.335633 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:03.335557    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:59:03.336357 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:03.336337    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 13:59:03.547027 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:03.546977    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx" event={"ID":"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4","Type":"ContainerStarted","Data":"2121ea35e202aeeed604bf63787c95326a0aac9f5e16747042f4417553d59521"}
Nov 02 13:59:03.598956 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:03.598895    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx" podStartSLOduration=2.598875971 podStartE2EDuration="2.598875971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:03.596746301 +0000 UTC m=+934.348050741" watchObservedRunningTime="2026-11-02 13:59:03.598875971 +0000 UTC m=+934.350180410"
Nov 02 13:59:04.555665 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:04.555630    5034 generic.go:356] "Generic (PLEG): container finished" podID="f8f91fa8-40ca-4847-a4f2-1abd215ac6d4" containerID="2121ea35e202aeeed604bf63787c95326a0aac9f5e16747042f4417553d59521" exitCode=0
Nov 02 13:59:04.556261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:04.555712    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx" event={"ID":"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4","Type":"ContainerDied","Data":"2121ea35e202aeeed604bf63787c95326a0aac9f5e16747042f4417553d59521"}
Nov 02 13:59:05.711934 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.711904    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:05.760286 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.760258    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-xks-ca\") pod \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") "
Nov 02 13:59:05.760438 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.760407    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-kube-api-access-pnq8w\" (UniqueName: \"kubernetes.io/projected/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-kube-api-access-pnq8w\") pod \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") "
Nov 02 13:59:05.760512 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.760485    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-openshift-ca\") pod \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\" (UID: \"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4\") "
Nov 02 13:59:05.761635 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.761594    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-openshift-ca" pod "f8f91fa8-40ca-4847-a4f2-1abd215ac6d4" (UID: "f8f91fa8-40ca-4847-a4f2-1abd215ac6d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:05.764645 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.764621    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-xks-ca" pod "f8f91fa8-40ca-4847-a4f2-1abd215ac6d4" (UID: "f8f91fa8-40ca-4847-a4f2-1abd215ac6d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:05.765150 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.765069    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-kube-api-access-pnq8w" pod "f8f91fa8-40ca-4847-a4f2-1abd215ac6d4" (UID: "f8f91fa8-40ca-4847-a4f2-1abd215ac6d4"). InnerVolumeSpecName "kube-api-access-pnq8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:05.861718 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.861629    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnq8w\" (UniqueName: \"kubernetes.io/projected/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-kube-api-access-pnq8w\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:05.861718 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.861666    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:05.861718 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.861678    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:05.932812 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.932770    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"]
Nov 02 13:59:05.933843 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.933818    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f8f91fa8-40ca-4847-a4f2-1abd215ac6d4" containerName="revoke-keys"
Nov 02 13:59:05.933843 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.933842    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f91fa8-40ca-4847-a4f2-1abd215ac6d4" containerName="revoke-keys"
Nov 02 13:59:05.934004 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.933970    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f91fa8-40ca-4847-a4f2-1abd215ac6d4" containerName="revoke-keys"
Nov 02 13:59:05.940064 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.940045    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:05.962258 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.962202    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad42e0\"/\"distinct-w4-ad42e0-kserve-self-signed-certs\""
Nov 02 13:59:05.963035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.962751    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"]
Nov 02 13:59:05.963035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.962835    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tmp-dir\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:05.963035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.962878    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-dshm\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:05.963035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.962962    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-model-cache\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:05.963035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.962994    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-home\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:05.963035 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.963036    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tls-certs\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:05.963394 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:05.963101    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbl9\" (UniqueName: \"kubernetes.io/projected/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-kube-api-access-wnbl9\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.064026 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.063996    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-dshm\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.064172 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.064151    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-model-cache\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.064229 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.064180    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-home\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.064229 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.064210    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tls-certs\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.064346 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.064283    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbl9\" (UniqueName: \"kubernetes.io/projected/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-kube-api-access-wnbl9\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.064468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.064404    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tmp-dir\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.064575 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.064553    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-home\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.064649 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.064603    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-model-cache\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.064798 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.064744    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tmp-dir\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.066289 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.066262    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-dshm\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.066767 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.066748    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tls-certs\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.088450 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.088427    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbl9\" (UniqueName: \"kubernetes.io/projected/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-kube-api-access-wnbl9\") pod \"distinct-w4-ad42e0-kserve-7665748d55-lwg7n\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.257551 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.257520    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:06.571627 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.571549    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"
Nov 02 13:59:06.571627 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.571575    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx" event={"ID":"f8f91fa8-40ca-4847-a4f2-1abd215ac6d4","Type":"ContainerDied","Data":"67a44842c4840aacd6cc1a74df7466c06a6b3198c8c4505b393f966a4a01c7e4"}
Nov 02 13:59:06.571827 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.571627    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67a44842c4840aacd6cc1a74df7466c06a6b3198c8c4505b393f966a4a01c7e4"
Nov 02 13:59:06.587420 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:06.587395    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbd677a1_311f_40ab_a9e8_f6c8c7c27ab1.slice/crio-c39c281994c35ed6d1c16519b4e109cb53dfc0aed1e03f00e0cf438412b5fd2c WatchSource:0}: Error finding container c39c281994c35ed6d1c16519b4e109cb53dfc0aed1e03f00e0cf438412b5fd2c: Status 404 returned error can't find the container with id c39c281994c35ed6d1c16519b4e109cb53dfc0aed1e03f00e0cf438412b5fd2c
Nov 02 13:59:06.588567 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:06.588543    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"]
Nov 02 13:59:07.579908 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:07.579875    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" event={"ID":"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1","Type":"ContainerStarted","Data":"59da64a98b1eb70ae7857d1ad90234ec8480fdb0da26c219c62acf9cfc0c641c"}
Nov 02 13:59:07.580319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:07.579918    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" event={"ID":"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1","Type":"ContainerStarted","Data":"c39c281994c35ed6d1c16519b4e109cb53dfc0aed1e03f00e0cf438412b5fd2c"}
Nov 02 13:59:07.626541 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:07.626493    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podStartSLOduration=2.626479425 podStartE2EDuration="2.626479425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:07.624672655 +0000 UTC m=+938.375977094" watchObservedRunningTime="2026-11-02 13:59:07.626479425 +0000 UTC m=+938.377783863"
Nov 02 13:59:08.257805 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:08.257741    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:12.683405 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.683322    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"]
Nov 02 13:59:12.687624 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.687600    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.724559 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.724526    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"]
Nov 02 13:59:12.750296 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.750266    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.750484 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.750312    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93258802-5287-409e-b9fb-71d0848c71c9-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.750484 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.750347    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.750484 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.750475    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.750600 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.750527    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.750600 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.750547    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93258802-5287-409e-b9fb-71d0848c71c9-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.750600 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.750565    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.750696 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.750671    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjqn\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-kube-api-access-8vjqn\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.750769 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.750748    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.852201 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852167    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.852384 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852213    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93258802-5287-409e-b9fb-71d0848c71c9-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.852384 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852247    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.852384 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852303    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.852384 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852362    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.852608 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852390    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93258802-5287-409e-b9fb-71d0848c71c9-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.852608 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852416    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.852608 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852472    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjqn\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-kube-api-access-8vjqn\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.852608 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852520    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.853020 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.852996    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-certs\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.853191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.853018    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-data\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.853273 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.853105    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.853273 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.853121    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-credential-socket\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.853626 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.853599    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93258802-5287-409e-b9fb-71d0848c71c9-istiod-ca-cert\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.855747 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.855437    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-envoy\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.855924 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.855896    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93258802-5287-409e-b9fb-71d0848c71c9-istio-podinfo\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.877351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.877310    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjqn\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-kube-api-access-8vjqn\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:12.877507 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:12.877464    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-istio-token\") pod \"e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") " pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:13.003819 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:13.003768    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:13.399759 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:13.399724    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"]
Nov 02 13:59:13.400193 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:13.400168    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93258802_5287_409e_b9fb_71d0848c71c9.slice/crio-1989584727c7608970cf24c2346988bf95e2f6f81b45bd36cc9dab90e8620575 WatchSource:0}: Error finding container 1989584727c7608970cf24c2346988bf95e2f6f81b45bd36cc9dab90e8620575: Status 404 returned error can't find the container with id 1989584727c7608970cf24c2346988bf95e2f6f81b45bd36cc9dab90e8620575
Nov 02 13:59:13.402769 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:13.402735    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:13.402912 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:13.402831    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:13.402912 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:13.402900    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:13.639315 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:13.635746    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" event={"ID":"93258802-5287-409e-b9fb-71d0848c71c9","Type":"ContainerStarted","Data":"7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f"}
Nov 02 13:59:13.639315 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:13.635818    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" event={"ID":"93258802-5287-409e-b9fb-71d0848c71c9","Type":"ContainerStarted","Data":"1989584727c7608970cf24c2346988bf95e2f6f81b45bd36cc9dab90e8620575"}
Nov 02 13:59:13.712855 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:13.710066    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podStartSLOduration=1.7100500950000002 podStartE2EDuration="1.710050095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:13.709448015 +0000 UTC m=+944.460752447" watchObservedRunningTime="2026-11-02 13:59:13.710050095 +0000 UTC m=+944.461354533"
Nov 02 13:59:14.005796 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:14.005516    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:14.013395 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:14.007041    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 13:59:14.013395 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:14.007093    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 13:59:15.018800 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:15.008092    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 13:59:15.018800 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:15.008173    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 13:59:16.004329 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.004292    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 13:59:16.004531 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.004355    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 13:59:16.258704 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.258610    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:16.300574 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.300343    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:16.629466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.629390    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"]
Nov 02 13:59:16.633541 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.633521    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:16.649810 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.649753    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:16.649810 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.649773    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:59:16.649972 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.649832    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:59:16.649972 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.649886    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:59:16.652663 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.652640    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"]
Nov 02 13:59:16.716232 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.716201    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:16.827741 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.827691    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02122b53-lc964\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:16.827929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.827794    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq56p\" (UniqueName: \"kubernetes.io/projected/ff39578a-fe15-4580-8f77-430c4168f96c-kube-api-access-kq56p\") pod \"maas-api-revoke-sub-02122b53-lc964\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:16.827929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.827893    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02122b53-lc964\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:16.928919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.928836    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02122b53-lc964\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:16.928919 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.928890    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq56p\" (UniqueName: \"kubernetes.io/projected/ff39578a-fe15-4580-8f77-430c4168f96c-kube-api-access-kq56p\") pod \"maas-api-revoke-sub-02122b53-lc964\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:16.929158 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.928939    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02122b53-lc964\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:16.929716 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.929694    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02122b53-lc964\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:16.931613 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.931589    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02122b53-lc964\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:16.953702 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:16.953677    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq56p\" (UniqueName: \"kubernetes.io/projected/ff39578a-fe15-4580-8f77-430c4168f96c-kube-api-access-kq56p\") pod \"maas-api-revoke-sub-02122b53-lc964\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:17.004852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:17.004718    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 13:59:17.004852 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:17.004822    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 13:59:17.248118 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:17.248090    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:17.606894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:17.606865    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"]
Nov 02 13:59:17.607377 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:17.607349    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff39578a_fe15_4580_8f77_430c4168f96c.slice/crio-b1e9e192ecc56b7da4275ceeb96ef815cb91871b91a5e733babad3808272a758 WatchSource:0}: Error finding container b1e9e192ecc56b7da4275ceeb96ef815cb91871b91a5e733babad3808272a758: Status 404 returned error can't find the container with id b1e9e192ecc56b7da4275ceeb96ef815cb91871b91a5e733babad3808272a758
Nov 02 13:59:17.709958 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:17.709861    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964" event={"ID":"ff39578a-fe15-4580-8f77-430c4168f96c","Type":"ContainerStarted","Data":"b1e9e192ecc56b7da4275ceeb96ef815cb91871b91a5e733babad3808272a758"}
Nov 02 13:59:18.005221 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:18.005056    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 13:59:18.005221 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:18.005132    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 13:59:18.430745 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:59:18.430656    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:59:18.725288 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:18.725180    5034 generic.go:356] "Generic (PLEG): container finished" podID="ff39578a-fe15-4580-8f77-430c4168f96c" containerID="3ebe64f48cb7792f7ed878a50eb334faa40bae86627da2f5589533e6bef9af87" exitCode=0
Nov 02 13:59:18.725288 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:18.725230    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964" event={"ID":"ff39578a-fe15-4580-8f77-430c4168f96c","Type":"ContainerDied","Data":"3ebe64f48cb7792f7ed878a50eb334faa40bae86627da2f5589533e6bef9af87"}
Nov 02 13:59:19.005799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.004220    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 13:59:19.005799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.004289    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 13:59:19.445280 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.445197    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"]
Nov 02 13:59:19.451150 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.451035    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.455222 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.455065    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"]
Nov 02 13:59:19.603358 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.602420    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-979b04d1-hhlwl\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.603358 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.602548    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-979b04d1-hhlwl\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.603358 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.602576    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxgq8\" (UniqueName: \"kubernetes.io/projected/e43f5b71-686d-4e91-b0be-43eb9c014584-kube-api-access-xxgq8\") pod \"maas-api-revoke-sub-979b04d1-hhlwl\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.711157 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.710876    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-979b04d1-hhlwl\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.712354 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.712327    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxgq8\" (UniqueName: \"kubernetes.io/projected/e43f5b71-686d-4e91-b0be-43eb9c014584-kube-api-access-xxgq8\") pod \"maas-api-revoke-sub-979b04d1-hhlwl\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.713062 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.713039    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-979b04d1-hhlwl\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.713733 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.712266    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-979b04d1-hhlwl\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.738249 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.737233    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-979b04d1-hhlwl\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.749773 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.748456    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxgq8\" (UniqueName: \"kubernetes.io/projected/e43f5b71-686d-4e91-b0be-43eb9c014584-kube-api-access-xxgq8\") pod \"maas-api-revoke-sub-979b04d1-hhlwl\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:19.780183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:19.780147    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:20.004193 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.004068    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 02 13:59:20.004535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.004510    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 02 13:59:20.038843 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.038821    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:20.112549 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.112407    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"]
Nov 02 13:59:20.131823 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.131787    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-xks-ca\") pod \"ff39578a-fe15-4580-8f77-430c4168f96c\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") "
Nov 02 13:59:20.131955 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.131935    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-openshift-ca\") pod \"ff39578a-fe15-4580-8f77-430c4168f96c\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") "
Nov 02 13:59:20.132021 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.131999    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff39578a-fe15-4580-8f77-430c4168f96c-kube-api-access-kq56p\" (UniqueName: \"kubernetes.io/projected/ff39578a-fe15-4580-8f77-430c4168f96c-kube-api-access-kq56p\") pod \"ff39578a-fe15-4580-8f77-430c4168f96c\" (UID: \"ff39578a-fe15-4580-8f77-430c4168f96c\") "
Nov 02 13:59:20.132440 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.132412    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-openshift-ca" pod "ff39578a-fe15-4580-8f77-430c4168f96c" (UID: "ff39578a-fe15-4580-8f77-430c4168f96c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:20.132599 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.132579    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:20.142969 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.142947    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-xks-ca" pod "ff39578a-fe15-4580-8f77-430c4168f96c" (UID: "ff39578a-fe15-4580-8f77-430c4168f96c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:20.144603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.144555    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff39578a-fe15-4580-8f77-430c4168f96c-kube-api-access-kq56p" pod "ff39578a-fe15-4580-8f77-430c4168f96c" (UID: "ff39578a-fe15-4580-8f77-430c4168f96c"). InnerVolumeSpecName "kube-api-access-kq56p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:20.235804 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.234295    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff39578a-fe15-4580-8f77-430c4168f96c-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:20.235804 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.234328    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq56p\" (UniqueName: \"kubernetes.io/projected/ff39578a-fe15-4580-8f77-430c4168f96c-kube-api-access-kq56p\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:20.797210 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.797177    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl" event={"ID":"e43f5b71-686d-4e91-b0be-43eb9c014584","Type":"ContainerStarted","Data":"7e23aa254c0f17ee7858ec55a3606367742dddc8e227c43964761eda0c8c377b"}
Nov 02 13:59:20.797882 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.797859    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl" event={"ID":"e43f5b71-686d-4e91-b0be-43eb9c014584","Type":"ContainerStarted","Data":"4ad0afd6a34686ccb3a4481eb10732737a103a91e95e5bf125f331850aaef96f"}
Nov 02 13:59:20.799433 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.799411    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"]
Nov 02 13:59:20.808128 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.808090    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ff39578a-fe15-4580-8f77-430c4168f96c" containerName="revoke-keys"
Nov 02 13:59:20.808128 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.808116    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff39578a-fe15-4580-8f77-430c4168f96c" containerName="revoke-keys"
Nov 02 13:59:20.808362 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.808259    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff39578a-fe15-4580-8f77-430c4168f96c" containerName="revoke-keys"
Nov 02 13:59:20.813737 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.810835    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"
Nov 02 13:59:20.828166 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.828139    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"]
Nov 02 13:59:20.828411 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.828372    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964" event={"ID":"ff39578a-fe15-4580-8f77-430c4168f96c","Type":"ContainerDied","Data":"b1e9e192ecc56b7da4275ceeb96ef815cb91871b91a5e733babad3808272a758"}
Nov 02 13:59:20.828675 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.828660    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1e9e192ecc56b7da4275ceeb96ef815cb91871b91a5e733babad3808272a758"
Nov 02 13:59:20.829645 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.828637    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:20.878748 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.878715    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b961700-rcjpx\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:20.878935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.878769    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b961700-rcjpx\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:20.878935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.878872    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4zj\" (UniqueName: \"kubernetes.io/projected/badb60d3-8f91-4126-8f5f-9a3dd2238036-kube-api-access-rh4zj\") pod \"maas-api-revoke-sub-8b961700-rcjpx\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:20.902309 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.900521    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl" podStartSLOduration=1.9005061840000002 podStartE2EDuration="1.900506184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:20.899684221 +0000 UTC m=+951.650988663" watchObservedRunningTime="2026-11-02 13:59:20.900506184 +0000 UTC m=+951.651810622"
Nov 02 13:59:20.980080 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.980043    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b961700-rcjpx\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:20.980260 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.980103    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b961700-rcjpx\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:20.980260 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.980210    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4zj\" (UniqueName: \"kubernetes.io/projected/badb60d3-8f91-4126-8f5f-9a3dd2238036-kube-api-access-rh4zj\") pod \"maas-api-revoke-sub-8b961700-rcjpx\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:20.980975 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.980952    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b961700-rcjpx\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:20.982588 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:20.982558    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b961700-rcjpx\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:21.008087 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.008019    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4zj\" (UniqueName: \"kubernetes.io/projected/badb60d3-8f91-4126-8f5f-9a3dd2238036-kube-api-access-rh4zj\") pod \"maas-api-revoke-sub-8b961700-rcjpx\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:21.150149 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.150101    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:21.329714 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.329549    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:21.330199 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.330181    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:21.331170 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.331137    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 13:59:21.503571 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:21.503546    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadb60d3_8f91_4126_8f5f_9a3dd2238036.slice/crio-d8eaf7c976aeb8dcfb0b3a86d727901c2e85ec773fa7d5a21688965e51b0ef99 WatchSource:0}: Error finding container d8eaf7c976aeb8dcfb0b3a86d727901c2e85ec773fa7d5a21688965e51b0ef99: Status 404 returned error can't find the container with id d8eaf7c976aeb8dcfb0b3a86d727901c2e85ec773fa7d5a21688965e51b0ef99
Nov 02 13:59:21.512433 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.512406    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"]
Nov 02 13:59:21.823872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.823412    5034 generic.go:356] "Generic (PLEG): container finished" podID="e43f5b71-686d-4e91-b0be-43eb9c014584" containerID="7e23aa254c0f17ee7858ec55a3606367742dddc8e227c43964761eda0c8c377b" exitCode=0
Nov 02 13:59:21.823872 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.823528    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl" event={"ID":"e43f5b71-686d-4e91-b0be-43eb9c014584","Type":"ContainerDied","Data":"7e23aa254c0f17ee7858ec55a3606367742dddc8e227c43964761eda0c8c377b"}
Nov 02 13:59:21.826211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.826185    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx" event={"ID":"badb60d3-8f91-4126-8f5f-9a3dd2238036","Type":"ContainerStarted","Data":"e58c7470acaad739a09170aae9ba32e35d3a448bc2fe243b7b5456fa5ae3ea8b"}
Nov 02 13:59:21.826286 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.826233    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx" event={"ID":"badb60d3-8f91-4126-8f5f-9a3dd2238036","Type":"ContainerStarted","Data":"d8eaf7c976aeb8dcfb0b3a86d727901c2e85ec773fa7d5a21688965e51b0ef99"}
Nov 02 13:59:21.911251 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:21.911200    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx" podStartSLOduration=1.911183471 podStartE2EDuration="1.911183471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:21.908501856 +0000 UTC m=+952.659806310" watchObservedRunningTime="2026-11-02 13:59:21.911183471 +0000 UTC m=+952.662487910"
Nov 02 13:59:22.834991 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:22.834908    5034 generic.go:356] "Generic (PLEG): container finished" podID="badb60d3-8f91-4126-8f5f-9a3dd2238036" containerID="e58c7470acaad739a09170aae9ba32e35d3a448bc2fe243b7b5456fa5ae3ea8b" exitCode=0
Nov 02 13:59:22.835456 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:22.834996    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx" event={"ID":"badb60d3-8f91-4126-8f5f-9a3dd2238036","Type":"ContainerDied","Data":"e58c7470acaad739a09170aae9ba32e35d3a448bc2fe243b7b5456fa5ae3ea8b"}
Nov 02 13:59:23.006250 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.006225    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:23.110505 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.110426    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-xks-ca\") pod \"e43f5b71-686d-4e91-b0be-43eb9c014584\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") "
Nov 02 13:59:23.110647 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.110615    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43f5b71-686d-4e91-b0be-43eb9c014584-kube-api-access-xxgq8\" (UniqueName: \"kubernetes.io/projected/e43f5b71-686d-4e91-b0be-43eb9c014584-kube-api-access-xxgq8\") pod \"e43f5b71-686d-4e91-b0be-43eb9c014584\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") "
Nov 02 13:59:23.110691 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.110659    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-openshift-ca\") pod \"e43f5b71-686d-4e91-b0be-43eb9c014584\" (UID: \"e43f5b71-686d-4e91-b0be-43eb9c014584\") "
Nov 02 13:59:23.111074 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.111048    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-openshift-ca" pod "e43f5b71-686d-4e91-b0be-43eb9c014584" (UID: "e43f5b71-686d-4e91-b0be-43eb9c014584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:23.111305 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.111290    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:23.113040 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.113011    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43f5b71-686d-4e91-b0be-43eb9c014584-kube-api-access-xxgq8" pod "e43f5b71-686d-4e91-b0be-43eb9c014584" (UID: "e43f5b71-686d-4e91-b0be-43eb9c014584"). InnerVolumeSpecName "kube-api-access-xxgq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:23.113040 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.113017    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-xks-ca" pod "e43f5b71-686d-4e91-b0be-43eb9c014584" (UID: "e43f5b71-686d-4e91-b0be-43eb9c014584"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:23.212874 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.212839    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e43f5b71-686d-4e91-b0be-43eb9c014584-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:23.212874 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.212865    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxgq8\" (UniqueName: \"kubernetes.io/projected/e43f5b71-686d-4e91-b0be-43eb9c014584-kube-api-access-xxgq8\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:23.845602 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.845574    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"
Nov 02 13:59:23.846611 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.845589    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl" event={"ID":"e43f5b71-686d-4e91-b0be-43eb9c014584","Type":"ContainerDied","Data":"4ad0afd6a34686ccb3a4481eb10732737a103a91e95e5bf125f331850aaef96f"}
Nov 02 13:59:23.846611 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:23.846573    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ad0afd6a34686ccb3a4481eb10732737a103a91e95e5bf125f331850aaef96f"
Nov 02 13:59:24.013614 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.013590    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:24.129397 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.129332    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-xks-ca\") pod \"badb60d3-8f91-4126-8f5f-9a3dd2238036\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") "
Nov 02 13:59:24.129564 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.129543    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badb60d3-8f91-4126-8f5f-9a3dd2238036-kube-api-access-rh4zj\" (UniqueName: \"kubernetes.io/projected/badb60d3-8f91-4126-8f5f-9a3dd2238036-kube-api-access-rh4zj\") pod \"badb60d3-8f91-4126-8f5f-9a3dd2238036\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") "
Nov 02 13:59:24.129690 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.129673    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-openshift-ca\") pod \"badb60d3-8f91-4126-8f5f-9a3dd2238036\" (UID: \"badb60d3-8f91-4126-8f5f-9a3dd2238036\") "
Nov 02 13:59:24.130006 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.129983    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-openshift-ca" pod "badb60d3-8f91-4126-8f5f-9a3dd2238036" (UID: "badb60d3-8f91-4126-8f5f-9a3dd2238036"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:24.130191 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.130174    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:24.131437 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.131412    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-xks-ca" pod "badb60d3-8f91-4126-8f5f-9a3dd2238036" (UID: "badb60d3-8f91-4126-8f5f-9a3dd2238036"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:24.131531 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.131515    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badb60d3-8f91-4126-8f5f-9a3dd2238036-kube-api-access-rh4zj" pod "badb60d3-8f91-4126-8f5f-9a3dd2238036" (UID: "badb60d3-8f91-4126-8f5f-9a3dd2238036"). InnerVolumeSpecName "kube-api-access-rh4zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:24.230969 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.230946    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/badb60d3-8f91-4126-8f5f-9a3dd2238036-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:24.230969 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.230969    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh4zj\" (UniqueName: \"kubernetes.io/projected/badb60d3-8f91-4126-8f5f-9a3dd2238036-kube-api-access-rh4zj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:24.853738 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.853706    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"
Nov 02 13:59:24.853738 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.853723    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx" event={"ID":"badb60d3-8f91-4126-8f5f-9a3dd2238036","Type":"ContainerDied","Data":"d8eaf7c976aeb8dcfb0b3a86d727901c2e85ec773fa7d5a21688965e51b0ef99"}
Nov 02 13:59:24.854168 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:24.853771    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8eaf7c976aeb8dcfb0b3a86d727901c2e85ec773fa7d5a21688965e51b0ef99"
Nov 02 13:59:28.075126 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.075088    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"]
Nov 02 13:59:28.075741 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.075720    5034 cpu_manager.go:414] "containerMap: removing container" podUID="badb60d3-8f91-4126-8f5f-9a3dd2238036" containerName="revoke-keys"
Nov 02 13:59:28.075741 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.075743    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badb60d3-8f91-4126-8f5f-9a3dd2238036" containerName="revoke-keys"
Nov 02 13:59:28.075925 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.075813    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e43f5b71-686d-4e91-b0be-43eb9c014584" containerName="revoke-keys"
Nov 02 13:59:28.075925 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.075823    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43f5b71-686d-4e91-b0be-43eb9c014584" containerName="revoke-keys"
Nov 02 13:59:28.076027 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.075997    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="badb60d3-8f91-4126-8f5f-9a3dd2238036" containerName="revoke-keys"
Nov 02 13:59:28.076027 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.076016    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43f5b71-686d-4e91-b0be-43eb9c014584" containerName="revoke-keys"
Nov 02 13:59:28.079822 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.079798    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.117967 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.117937    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"]
Nov 02 13:59:28.173903 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.173870    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.174065 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.174041    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.174110 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.174084    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.174184 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.174163    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnm8w\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-kube-api-access-xnm8w\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.174357 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.174337    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b78b2a-6676-4572-9b89-7b8e60df80f6-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.174438 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.174379    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.174501 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.174484    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.174534 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.174525    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.174612 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.174594    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.275611 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.275579    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b78b2a-6676-4572-9b89-7b8e60df80f6-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.275759 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.275627    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.275759 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.275671    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.275759 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.275703    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.275835    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.275907    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276128 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.276012    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276128 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.276040    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276128 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.276079    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnm8w\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-kube-api-access-xnm8w\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276404 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.276374    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b78b2a-6676-4572-9b89-7b8e60df80f6-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276702 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.276677    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276850 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.276719    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276850 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.276755    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.276850 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.276773    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.278895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.278874    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.278895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.278887    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.300345 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.300320    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnm8w\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-kube-api-access-xnm8w\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.300738 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.300718    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.398181 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.398112    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:28.728674 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:28.728642    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58b78b2a_6676_4572_9b89_7b8e60df80f6.slice/crio-0a34b0e1286ac4be3e29f39d4b4d565433ca52df8f7d7baf196cd95ac59e95ca WatchSource:0}: Error finding container 0a34b0e1286ac4be3e29f39d4b4d565433ca52df8f7d7baf196cd95ac59e95ca: Status 404 returned error can't find the container with id 0a34b0e1286ac4be3e29f39d4b4d565433ca52df8f7d7baf196cd95ac59e95ca
Nov 02 13:59:28.732211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.732181    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"]
Nov 02 13:59:28.747656 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.747535    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:28.748291 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.748254    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:28.748504 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.748472    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:28.890695 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.890651    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" event={"ID":"58b78b2a-6676-4572-9b89-7b8e60df80f6","Type":"ContainerStarted","Data":"db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df"}
Nov 02 13:59:28.890873 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.890728    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" event={"ID":"58b78b2a-6676-4572-9b89-7b8e60df80f6","Type":"ContainerStarted","Data":"0a34b0e1286ac4be3e29f39d4b4d565433ca52df8f7d7baf196cd95ac59e95ca"}
Nov 02 13:59:28.942744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:28.942693    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podStartSLOduration=0.942678757 podStartE2EDuration="942.678757ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:28.940870338 +0000 UTC m=+959.692174772" watchObservedRunningTime="2026-11-02 13:59:28.942678757 +0000 UTC m=+959.693983194"
Nov 02 13:59:29.405211 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:29.405177    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 13:59:29.405743 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:29.405712    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 13:59:29.410551 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:29.410520    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:30.403801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:30.400318    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 13:59:30.403801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:30.400397    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 13:59:31.404803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:31.404440    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 13:59:31.404803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:31.404497    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 13:59:31.866802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:31.865847    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 13:59:31.912802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:31.909859    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.071803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.070867    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.171867 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.169900    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.181799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.175852    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.217971 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.217915    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.401190 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.401154    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 13:59:32.401475 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.401451    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 13:59:32.671801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.670850    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.695810 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.693031    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.867817 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.864851    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.867817 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.866625    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.867817 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.866999    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:59:32.915813 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.913951    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:32.915813 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:32.914337    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:59:33.365820 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:33.361758    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 13:59:33.373802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:33.366927    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:33.373802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:33.370400    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 13:59:33.373802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:33.370946    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 13:59:33.395500 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:33.395469    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:59:33.421797 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:33.406100    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 13:59:33.421797 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:33.406154    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 13:59:33.602087 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:33.602046    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:33.704851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:33.694870    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:34.011915 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.011878    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 13:59:34.022069 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.022031    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:59:34.022809 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.022273    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:59:34.023054 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.023037    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:34.026894 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.026869    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 13:59:34.027068 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.027055    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 13:59:34.061826 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.050671    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:34.093976 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.093950    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 13:59:34.154762 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.154725    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 13:59:34.157464 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.157440    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 13:59:34.158420 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.158402    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 13:59:34.409433 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.409349    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 13:59:34.409589 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.409424    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 13:59:34.568237 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.568207    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"]
Nov 02 13:59:34.599453 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.596411    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.599631 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.599511    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"]
Nov 02 13:59:34.718249 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.718154    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.718665 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.718285    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.718665 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.718368    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.718665 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.718445    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.718665 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.718526    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.718665 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.718572    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.718665 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.718636    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.718930 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.718689    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.718930 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.718705    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5n5g\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-kube-api-access-n5n5g\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.823157 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.823115    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.823319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.823169    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5n5g\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-kube-api-access-n5n5g\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.823319 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.823273    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.823400 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.823387    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.823438 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.823425    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.823482 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.823472    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.823550 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.823535    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.823611 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.823597    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.823682 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.823666    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.824311 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.824241    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.824738 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.824612    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.824861 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.824837    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.825107 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.825061    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.825107 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.825097    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.826236 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.826209    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.827377 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.827354    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.848071 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.848043    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.848763 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.848741    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5n5g\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-kube-api-access-n5n5g\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:34.915851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:34.915816    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:35.123186 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.123156    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"]
Nov 02 13:59:35.128421 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.128400    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.153638 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.153608    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:59:35.155037 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.155006    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:59:35.158826 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.157041    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"]
Nov 02 13:59:35.159062 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.157179    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:59:35.159179 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.157234    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:35.228183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.228096    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62d126b4-tt5gk\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.228344 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.228222    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62d126b4-tt5gk\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.228432 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.228348    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9lw\" (UniqueName: \"kubernetes.io/projected/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-kube-api-access-gj9lw\") pod \"maas-api-revoke-sub-62d126b4-tt5gk\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.263922 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:35.263897    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ce2dc0_9f5e_47e8_970b_d407bb0036e3.slice/crio-d10f9a062b3bf439d95e28848813cae8362eb57f75b32fdf569351e1463aaa73 WatchSource:0}: Error finding container d10f9a062b3bf439d95e28848813cae8362eb57f75b32fdf569351e1463aaa73: Status 404 returned error can't find the container with id d10f9a062b3bf439d95e28848813cae8362eb57f75b32fdf569351e1463aaa73
Nov 02 13:59:35.266945 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.266910    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:35.267057 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.266976    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:35.267057 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.267008    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:35.268261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.268237    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"]
Nov 02 13:59:35.330106 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.330074    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62d126b4-tt5gk\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.330221 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.330160    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62d126b4-tt5gk\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.330284 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.330254    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9lw\" (UniqueName: \"kubernetes.io/projected/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-kube-api-access-gj9lw\") pod \"maas-api-revoke-sub-62d126b4-tt5gk\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.330751 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.330730    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62d126b4-tt5gk\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.332927 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.332860    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62d126b4-tt5gk\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.354634 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.354614    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9lw\" (UniqueName: \"kubernetes.io/projected/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-kube-api-access-gj9lw\") pod \"maas-api-revoke-sub-62d126b4-tt5gk\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.399799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.398818    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 13:59:35.399799 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.398871    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 13:59:35.420629 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.420592    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" event={"ID":"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3","Type":"ContainerStarted","Data":"63b9828b5a33e79040d344a8d772fc9c1b062e7e3a2c28514145d16c5948a968"}
Nov 02 13:59:35.420763 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.420659    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" event={"ID":"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3","Type":"ContainerStarted","Data":"d10f9a062b3bf439d95e28848813cae8362eb57f75b32fdf569351e1463aaa73"}
Nov 02 13:59:35.443142 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.443109    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:35.505753 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.505597    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" podStartSLOduration=1.505578839 podStartE2EDuration="1.505578839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:35.500932053 +0000 UTC m=+966.252236492" watchObservedRunningTime="2026-11-02 13:59:35.505578839 +0000 UTC m=+966.256883277"
Nov 02 13:59:35.805427 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.805343    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"]
Nov 02 13:59:35.809804 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:35.809758    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f2f1cf_d0ce_4c0f_ba95_8a1898374b6a.slice/crio-cb53b58733c1323b1cbb126a47935304a2982162567921c21f2e0bf69989a2d9 WatchSource:0}: Error finding container cb53b58733c1323b1cbb126a47935304a2982162567921c21f2e0bf69989a2d9: Status 404 returned error can't find the container with id cb53b58733c1323b1cbb126a47935304a2982162567921c21f2e0bf69989a2d9
Nov 02 13:59:35.917047 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:35.916948    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:36.400205 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.400114    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 02 13:59:36.400446 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.400419    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 02 13:59:36.431567 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.431532    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk" event={"ID":"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a","Type":"ContainerStarted","Data":"4f597b63f04b5f0b7746b99009e514727428f33065034aa7cec67b093e56fdcb"}
Nov 02 13:59:36.431807 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.431591    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk" event={"ID":"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a","Type":"ContainerStarted","Data":"cb53b58733c1323b1cbb126a47935304a2982162567921c21f2e0bf69989a2d9"}
Nov 02 13:59:36.479801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.477158    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk" podStartSLOduration=1.477136415 podStartE2EDuration="1.477136415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:36.475179725 +0000 UTC m=+967.226484165" watchObservedRunningTime="2026-11-02 13:59:36.477136415 +0000 UTC m=+967.228440855"
Nov 02 13:59:36.508647 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.508618    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"]
Nov 02 13:59:36.513256 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.513227    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.519001 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.518968    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"]
Nov 02 13:59:36.563797 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.559662    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c54e268-wnsk9\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.563797 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.559759    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c54e268-wnsk9\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.563797 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.559862    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6vj4\" (UniqueName: \"kubernetes.io/projected/aea1ff31-bc92-4d49-82ba-18924cbe8664-kube-api-access-m6vj4\") pod \"maas-api-revoke-sub-9c54e268-wnsk9\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.661168 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.661089    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c54e268-wnsk9\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.661168 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.661156    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c54e268-wnsk9\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.661359 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.661219    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6vj4\" (UniqueName: \"kubernetes.io/projected/aea1ff31-bc92-4d49-82ba-18924cbe8664-kube-api-access-m6vj4\") pod \"maas-api-revoke-sub-9c54e268-wnsk9\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.661905 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.661885    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c54e268-wnsk9\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.663634 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.663603    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c54e268-wnsk9\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.703545 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.703506    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6vj4\" (UniqueName: \"kubernetes.io/projected/aea1ff31-bc92-4d49-82ba-18924cbe8664-kube-api-access-m6vj4\") pod \"maas-api-revoke-sub-9c54e268-wnsk9\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.832557 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.832522    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:36.920507 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.920482    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 13:59:36.920882 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:36.920678    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 13:59:37.204200 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:37.196895    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"]
Nov 02 13:59:37.228949 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:37.228922    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea1ff31_bc92_4d49_82ba_18924cbe8664.slice/crio-f3c56c09c3e903cf3c82c472f272f033b055a77b0eeb3dfcea53af01c7a2dddf WatchSource:0}: Error finding container f3c56c09c3e903cf3c82c472f272f033b055a77b0eeb3dfcea53af01c7a2dddf: Status 404 returned error can't find the container with id f3c56c09c3e903cf3c82c472f272f033b055a77b0eeb3dfcea53af01c7a2dddf
Nov 02 13:59:37.443539 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:37.443502    5034 generic.go:356] "Generic (PLEG): container finished" podID="42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a" containerID="4f597b63f04b5f0b7746b99009e514727428f33065034aa7cec67b093e56fdcb" exitCode=0
Nov 02 13:59:37.443735 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:37.443593    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk" event={"ID":"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a","Type":"ContainerDied","Data":"4f597b63f04b5f0b7746b99009e514727428f33065034aa7cec67b093e56fdcb"}
Nov 02 13:59:37.445056 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:37.445030    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9" event={"ID":"aea1ff31-bc92-4d49-82ba-18924cbe8664","Type":"ContainerStarted","Data":"f3c56c09c3e903cf3c82c472f272f033b055a77b0eeb3dfcea53af01c7a2dddf"}
Nov 02 13:59:37.557875 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:37.557711    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:37.558651 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:37.558578    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:37.559528 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:37.559466    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 13:59:37.924863 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:37.922546    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 13:59:37.924863 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:37.922610    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 13:59:38.454526 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.454498    5034 generic.go:356] "Generic (PLEG): container finished" podID="aea1ff31-bc92-4d49-82ba-18924cbe8664" containerID="edc64545a49cb34cbb8dd1b76a415cf670bceceede0113eb4a249990351cda88" exitCode=0
Nov 02 13:59:38.454649 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.454584    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9" event={"ID":"aea1ff31-bc92-4d49-82ba-18924cbe8664","Type":"ContainerDied","Data":"edc64545a49cb34cbb8dd1b76a415cf670bceceede0113eb4a249990351cda88"}
Nov 02 13:59:38.743571 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.743182    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:38.848352 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.848317    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-xks-ca\") pod \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") "
Nov 02 13:59:38.849049 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.849024    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-openshift-ca\") pod \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") "
Nov 02 13:59:38.849264 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.849245    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-kube-api-access-gj9lw\" (UniqueName: \"kubernetes.io/projected/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-kube-api-access-gj9lw\") pod \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\" (UID: \"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a\") "
Nov 02 13:59:38.852300 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.852282    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-openshift-ca" pod "42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a" (UID: "42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:38.854306 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.854288    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-xks-ca" pod "42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a" (UID: "42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:38.854506 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.854493    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-kube-api-access-gj9lw" pod "42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a" (UID: "42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a"). InnerVolumeSpecName "kube-api-access-gj9lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:38.917181 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.917150    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 13:59:38.927911 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.927856    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 13:59:38.951895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.951815    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9lw\" (UniqueName: \"kubernetes.io/projected/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-kube-api-access-gj9lw\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:38.951895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.951849    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:38.951895 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:38.951859    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:39.167100 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.166121    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"]
Nov 02 13:59:39.167100 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.166987    5034 cpu_manager.go:414] "containerMap: removing container" podUID="42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a" containerName="revoke-keys"
Nov 02 13:59:39.167100 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.167004    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a" containerName="revoke-keys"
Nov 02 13:59:39.167330 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.167204    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a" containerName="revoke-keys"
Nov 02 13:59:39.197801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.197073    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.223039 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.223003    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"]
Nov 02 13:59:39.358848 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.358813    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdl5t\" (UniqueName: \"kubernetes.io/projected/b5dc46cd-79c7-4598-9ede-b2c594efc915-kube-api-access-bdl5t\") pod \"maas-api-revoke-sub-6d62804e-mqnm9\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.359008 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.358863    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d62804e-mqnm9\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.359125 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.359096    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d62804e-mqnm9\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.460447 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.460417    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdl5t\" (UniqueName: \"kubernetes.io/projected/b5dc46cd-79c7-4598-9ede-b2c594efc915-kube-api-access-bdl5t\") pod \"maas-api-revoke-sub-6d62804e-mqnm9\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.460623 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.460464    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d62804e-mqnm9\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.460623 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.460557    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d62804e-mqnm9\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.461251 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.461227    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d62804e-mqnm9\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.462862 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.462843    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d62804e-mqnm9\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.464342 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.464320    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"
Nov 02 13:59:39.464713 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.464696    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk" event={"ID":"42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a","Type":"ContainerDied","Data":"cb53b58733c1323b1cbb126a47935304a2982162567921c21f2e0bf69989a2d9"}
Nov 02 13:59:39.464833 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.464729    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb53b58733c1323b1cbb126a47935304a2982162567921c21f2e0bf69989a2d9"
Nov 02 13:59:39.547877 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.547831    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdl5t\" (UniqueName: \"kubernetes.io/projected/b5dc46cd-79c7-4598-9ede-b2c594efc915-kube-api-access-bdl5t\") pod \"maas-api-revoke-sub-6d62804e-mqnm9\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.749484 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.749461    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:39.825840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.825741    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:39.867667 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.866746    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-openshift-ca\") pod \"aea1ff31-bc92-4d49-82ba-18924cbe8664\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") "
Nov 02 13:59:39.867667 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.867135    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-openshift-ca" pod "aea1ff31-bc92-4d49-82ba-18924cbe8664" (UID: "aea1ff31-bc92-4d49-82ba-18924cbe8664"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:39.867667 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.867309    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-xks-ca\") pod \"aea1ff31-bc92-4d49-82ba-18924cbe8664\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") "
Nov 02 13:59:39.867941 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.867742    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aea1ff31-bc92-4d49-82ba-18924cbe8664-kube-api-access-m6vj4\" (UniqueName: \"kubernetes.io/projected/aea1ff31-bc92-4d49-82ba-18924cbe8664-kube-api-access-m6vj4\") pod \"aea1ff31-bc92-4d49-82ba-18924cbe8664\" (UID: \"aea1ff31-bc92-4d49-82ba-18924cbe8664\") "
Nov 02 13:59:39.868358 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.868331    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:39.887440 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.882907    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-xks-ca" pod "aea1ff31-bc92-4d49-82ba-18924cbe8664" (UID: "aea1ff31-bc92-4d49-82ba-18924cbe8664"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:39.908290 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.890730    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea1ff31-bc92-4d49-82ba-18924cbe8664-kube-api-access-m6vj4" pod "aea1ff31-bc92-4d49-82ba-18924cbe8664" (UID: "aea1ff31-bc92-4d49-82ba-18924cbe8664"). InnerVolumeSpecName "kube-api-access-m6vj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:39.920826 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.917429    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 13:59:39.920826 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.917484    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 13:59:39.990801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.969761    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aea1ff31-bc92-4d49-82ba-18924cbe8664-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:39.990801 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:39.969817    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6vj4\" (UniqueName: \"kubernetes.io/projected/aea1ff31-bc92-4d49-82ba-18924cbe8664-kube-api-access-m6vj4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:40.231322 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:40.230480    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"]
Nov 02 13:59:40.490802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:40.484676    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9" event={"ID":"b5dc46cd-79c7-4598-9ede-b2c594efc915","Type":"ContainerStarted","Data":"721fb9ed774d5b444abacd0d7e2dce930d14976b8c350fdf746e483a17fb8b5a"}
Nov 02 13:59:40.490802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:40.487168    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9" event={"ID":"aea1ff31-bc92-4d49-82ba-18924cbe8664","Type":"ContainerDied","Data":"f3c56c09c3e903cf3c82c472f272f033b055a77b0eeb3dfcea53af01c7a2dddf"}
Nov 02 13:59:40.490802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:40.487233    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3c56c09c3e903cf3c82c472f272f033b055a77b0eeb3dfcea53af01c7a2dddf"
Nov 02 13:59:40.490802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:40.487319    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"
Nov 02 13:59:40.917007 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:40.916973    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 13:59:40.917205 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:40.917028    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 13:59:41.502608 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:41.502569    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9" event={"ID":"b5dc46cd-79c7-4598-9ede-b2c594efc915","Type":"ContainerStarted","Data":"fad2e8ee50b96f9ff0e5743a666c7834514a4a390b38e98b1e297e7c4c500cff"}
Nov 02 13:59:41.545229 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:41.545180    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9" podStartSLOduration=2.54516577 podStartE2EDuration="2.54516577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:41.544369077 +0000 UTC m=+972.295673518" watchObservedRunningTime="2026-11-02 13:59:41.54516577 +0000 UTC m=+972.296470208"
Nov 02 13:59:41.916624 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:41.916546    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 02 13:59:41.916769 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:41.916613    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 02 13:59:42.513583 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:42.513543    5034 generic.go:356] "Generic (PLEG): container finished" podID="b5dc46cd-79c7-4598-9ede-b2c594efc915" containerID="fad2e8ee50b96f9ff0e5743a666c7834514a4a390b38e98b1e297e7c4c500cff" exitCode=0
Nov 02 13:59:42.513583 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:42.513590    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9" event={"ID":"b5dc46cd-79c7-4598-9ede-b2c594efc915","Type":"ContainerDied","Data":"fad2e8ee50b96f9ff0e5743a666c7834514a4a390b38e98b1e297e7c4c500cff"}
Nov 02 13:59:43.012578 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:43.012541    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:43.015225 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:43.014619    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:43.016962 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:43.016936    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 13:59:44.763041 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:44.763004    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:44.782964 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:44.764806    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9" event={"ID":"b5dc46cd-79c7-4598-9ede-b2c594efc915","Type":"ContainerDied","Data":"721fb9ed774d5b444abacd0d7e2dce930d14976b8c350fdf746e483a17fb8b5a"}
Nov 02 13:59:44.783241 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:44.783218    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721fb9ed774d5b444abacd0d7e2dce930d14976b8c350fdf746e483a17fb8b5a"
Nov 02 13:59:44.910376 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:44.910338    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-openshift-ca\") pod \"b5dc46cd-79c7-4598-9ede-b2c594efc915\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") "
Nov 02 13:59:44.910840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:44.910815    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-xks-ca\") pod \"b5dc46cd-79c7-4598-9ede-b2c594efc915\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") "
Nov 02 13:59:44.911142 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:44.911109    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5dc46cd-79c7-4598-9ede-b2c594efc915-kube-api-access-bdl5t\" (UniqueName: \"kubernetes.io/projected/b5dc46cd-79c7-4598-9ede-b2c594efc915-kube-api-access-bdl5t\") pod \"b5dc46cd-79c7-4598-9ede-b2c594efc915\" (UID: \"b5dc46cd-79c7-4598-9ede-b2c594efc915\") "
Nov 02 13:59:44.913823 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:44.913550    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-openshift-ca" pod "b5dc46cd-79c7-4598-9ede-b2c594efc915" (UID: "b5dc46cd-79c7-4598-9ede-b2c594efc915"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:44.968114 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:44.968055    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5dc46cd-79c7-4598-9ede-b2c594efc915-kube-api-access-bdl5t" pod "b5dc46cd-79c7-4598-9ede-b2c594efc915" (UID: "b5dc46cd-79c7-4598-9ede-b2c594efc915"). InnerVolumeSpecName "kube-api-access-bdl5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:44.968362 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:44.968266    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-xks-ca" pod "b5dc46cd-79c7-4598-9ede-b2c594efc915" (UID: "b5dc46cd-79c7-4598-9ede-b2c594efc915"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:45.021939 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.021866    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:45.021939 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.021893    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dc46cd-79c7-4598-9ede-b2c594efc915-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:45.021939 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.021908    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdl5t\" (UniqueName: \"kubernetes.io/projected/b5dc46cd-79c7-4598-9ede-b2c594efc915-kube-api-access-bdl5t\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:45.506431 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.506395    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"]
Nov 02 13:59:45.507287 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.507263    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b5dc46cd-79c7-4598-9ede-b2c594efc915" containerName="revoke-keys"
Nov 02 13:59:45.507287 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.507286    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5dc46cd-79c7-4598-9ede-b2c594efc915" containerName="revoke-keys"
Nov 02 13:59:45.507456 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.507352    5034 cpu_manager.go:414] "containerMap: removing container" podUID="aea1ff31-bc92-4d49-82ba-18924cbe8664" containerName="revoke-keys"
Nov 02 13:59:45.507456 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.507361    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aea1ff31-bc92-4d49-82ba-18924cbe8664" containerName="revoke-keys"
Nov 02 13:59:45.507558 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.507517    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5dc46cd-79c7-4598-9ede-b2c594efc915" containerName="revoke-keys"
Nov 02 13:59:45.507558 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.507538    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="aea1ff31-bc92-4d49-82ba-18924cbe8664" containerName="revoke-keys"
Nov 02 13:59:45.511703 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.511680    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.529293 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.529268    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"]
Nov 02 13:59:45.631337 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.631297    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5jb\" (UniqueName: \"kubernetes.io/projected/b9366740-9c2a-4df2-b1b1-e11f7beb2511-kube-api-access-ch5jb\") pod \"maas-api-revoke-sub-9fcc369c-c99vx\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.631484 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.631432    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fcc369c-c99vx\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.631484 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.631471    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fcc369c-c99vx\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.732339 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.732314    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fcc369c-c99vx\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.732465 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.732355    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fcc369c-c99vx\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.732521 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.732469    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5jb\" (UniqueName: \"kubernetes.io/projected/b9366740-9c2a-4df2-b1b1-e11f7beb2511-kube-api-access-ch5jb\") pod \"maas-api-revoke-sub-9fcc369c-c99vx\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.733011 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.732987    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fcc369c-c99vx\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.734691 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.734671    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fcc369c-c99vx\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.756658 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.756636    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5jb\" (UniqueName: \"kubernetes.io/projected/b9366740-9c2a-4df2-b1b1-e11f7beb2511-kube-api-access-ch5jb\") pod \"maas-api-revoke-sub-9fcc369c-c99vx\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:45.771209 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.771189    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"
Nov 02 13:59:45.825700 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:45.825678    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:46.135749 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:46.135720    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9366740_9c2a_4df2_b1b1_e11f7beb2511.slice/crio-41b1cea2fea7cb53455f80c184641e268529c39c98cb49da6c464d4600b818f7 WatchSource:0}: Error finding container 41b1cea2fea7cb53455f80c184641e268529c39c98cb49da6c464d4600b818f7: Status 404 returned error can't find the container with id 41b1cea2fea7cb53455f80c184641e268529c39c98cb49da6c464d4600b818f7
Nov 02 13:59:46.138385 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:46.138360    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"]
Nov 02 13:59:46.783175 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:46.783135    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx" event={"ID":"b9366740-9c2a-4df2-b1b1-e11f7beb2511","Type":"ContainerStarted","Data":"dc5c6766cfd331d624ebf15f615db171034ab4bb2c1cc535f29e1ca07932967a"}
Nov 02 13:59:46.783589 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:46.783204    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx" event={"ID":"b9366740-9c2a-4df2-b1b1-e11f7beb2511","Type":"ContainerStarted","Data":"41b1cea2fea7cb53455f80c184641e268529c39c98cb49da6c464d4600b818f7"}
Nov 02 13:59:46.871997 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:46.871938    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx" podStartSLOduration=1.871924368 podStartE2EDuration="1.871924368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:46.868112967 +0000 UTC m=+977.619417405" watchObservedRunningTime="2026-11-02 13:59:46.871924368 +0000 UTC m=+977.623228806"
Nov 02 13:59:47.024216 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.024183    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"]
Nov 02 13:59:47.028722 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.028699    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.045755 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.045726    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkwnt\" (UniqueName: \"kubernetes.io/projected/90beea4a-ca68-454f-95c9-d7576b8d8c8d-kube-api-access-qkwnt\") pod \"maas-api-revoke-sub-e9953549-9wn8s\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.045944 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.045919    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9953549-9wn8s\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.046173 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.046150    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9953549-9wn8s\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.049707 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.047904    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"]
Nov 02 13:59:47.147273 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.147241    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkwnt\" (UniqueName: \"kubernetes.io/projected/90beea4a-ca68-454f-95c9-d7576b8d8c8d-kube-api-access-qkwnt\") pod \"maas-api-revoke-sub-e9953549-9wn8s\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.147390 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.147354    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9953549-9wn8s\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.147468 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.147450    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9953549-9wn8s\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.148140 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.148115    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9953549-9wn8s\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.149703 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.149680    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9953549-9wn8s\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.174104 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.174079    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkwnt\" (UniqueName: \"kubernetes.io/projected/90beea4a-ca68-454f-95c9-d7576b8d8c8d-kube-api-access-qkwnt\") pod \"maas-api-revoke-sub-e9953549-9wn8s\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.344323 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.344256    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:47.648898 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:47.648865    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90beea4a_ca68_454f_95c9_d7576b8d8c8d.slice/crio-6c8041cab03f42a819f9db7f6b2ba4c8593eb1825581692d28eeffbe7f7aa9bf WatchSource:0}: Error finding container 6c8041cab03f42a819f9db7f6b2ba4c8593eb1825581692d28eeffbe7f7aa9bf: Status 404 returned error can't find the container with id 6c8041cab03f42a819f9db7f6b2ba4c8593eb1825581692d28eeffbe7f7aa9bf
Nov 02 13:59:47.650142 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.650117    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"]
Nov 02 13:59:47.792840 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.792771    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s" event={"ID":"90beea4a-ca68-454f-95c9-d7576b8d8c8d","Type":"ContainerStarted","Data":"6c8041cab03f42a819f9db7f6b2ba4c8593eb1825581692d28eeffbe7f7aa9bf"}
Nov 02 13:59:47.794519 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.794491    5034 generic.go:356] "Generic (PLEG): container finished" podID="b9366740-9c2a-4df2-b1b1-e11f7beb2511" containerID="dc5c6766cfd331d624ebf15f615db171034ab4bb2c1cc535f29e1ca07932967a" exitCode=0
Nov 02 13:59:47.794671 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:47.794539    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx" event={"ID":"b9366740-9c2a-4df2-b1b1-e11f7beb2511","Type":"ContainerDied","Data":"dc5c6766cfd331d624ebf15f615db171034ab4bb2c1cc535f29e1ca07932967a"}
Nov 02 13:59:48.413257 ip-10-0-139-26 kubenswrapper[5034]: E1102 13:59:48.413219    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 13:59:48.677643 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.677571    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"]
Nov 02 13:59:48.683772 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.683749    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.712196 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.712168    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-ad42e0\"/\"distinct-2-w4-ad42e0-kserve-self-signed-certs\""
Nov 02 13:59:48.716743 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.716720    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"]
Nov 02 13:59:48.771140 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.771109    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-tmp-dir\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.771305 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.771175    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-model-cache\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.771365 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.771304    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-home\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.771444 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.771421    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xng2w\" (UniqueName: \"kubernetes.io/projected/e06878a8-2080-4a80-a243-d37ec2ed7076-kube-api-access-xng2w\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.771504 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.771462    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-dshm\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.771504 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.771498    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06878a8-2080-4a80-a243-d37ec2ed7076-tls-certs\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.803870 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.803833    5034 generic.go:356] "Generic (PLEG): container finished" podID="90beea4a-ca68-454f-95c9-d7576b8d8c8d" containerID="de67e7d63ea223720a55597bbce41f628a181cf5acbd99938e3ed0a32b806844" exitCode=0
Nov 02 13:59:48.804336 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.803917    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s" event={"ID":"90beea4a-ca68-454f-95c9-d7576b8d8c8d","Type":"ContainerDied","Data":"de67e7d63ea223720a55597bbce41f628a181cf5acbd99938e3ed0a32b806844"}
Nov 02 13:59:48.904814 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.897757    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-home\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.904814 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.897898    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xng2w\" (UniqueName: \"kubernetes.io/projected/e06878a8-2080-4a80-a243-d37ec2ed7076-kube-api-access-xng2w\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.904814 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.897936    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-dshm\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.904814 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.897985    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06878a8-2080-4a80-a243-d37ec2ed7076-tls-certs\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.904814 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.898075    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-tmp-dir\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.924803 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.924249    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-tmp-dir\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.950226 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.944981    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-model-cache\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.950226 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.945473    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-model-cache\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.950226 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.945717    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-home\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.967057 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.967004    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-dshm\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.975193 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.974691    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06878a8-2080-4a80-a243-d37ec2ed7076-tls-certs\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.998667 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.979430    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xng2w\" (UniqueName: \"kubernetes.io/projected/e06878a8-2080-4a80-a243-d37ec2ed7076-kube-api-access-xng2w\") pod \"distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") " pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:48.998667 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:48.997022    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:49.051440 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.051189    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:49.147744 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.147713    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-openshift-ca\") pod \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") "
Nov 02 13:59:49.147932 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.147863    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9366740-9c2a-4df2-b1b1-e11f7beb2511-kube-api-access-ch5jb\" (UniqueName: \"kubernetes.io/projected/b9366740-9c2a-4df2-b1b1-e11f7beb2511-kube-api-access-ch5jb\") pod \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") "
Nov 02 13:59:49.147932 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.147915    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-xks-ca\") pod \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\" (UID: \"b9366740-9c2a-4df2-b1b1-e11f7beb2511\") "
Nov 02 13:59:49.148192 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.148162    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-openshift-ca" pod "b9366740-9c2a-4df2-b1b1-e11f7beb2511" (UID: "b9366740-9c2a-4df2-b1b1-e11f7beb2511"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:49.148513 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.148488    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:49.150128 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.150096    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-xks-ca" pod "b9366740-9c2a-4df2-b1b1-e11f7beb2511" (UID: "b9366740-9c2a-4df2-b1b1-e11f7beb2511"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:49.150218 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.150155    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9366740-9c2a-4df2-b1b1-e11f7beb2511-kube-api-access-ch5jb" pod "b9366740-9c2a-4df2-b1b1-e11f7beb2511" (UID: "b9366740-9c2a-4df2-b1b1-e11f7beb2511"). InnerVolumeSpecName "kube-api-access-ch5jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:49.250133 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.250107    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5jb\" (UniqueName: \"kubernetes.io/projected/b9366740-9c2a-4df2-b1b1-e11f7beb2511-kube-api-access-ch5jb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:49.250133 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.250132    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9366740-9c2a-4df2-b1b1-e11f7beb2511-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:49.317526 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.317491    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"]
Nov 02 13:59:49.319340 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:49.319309    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode06878a8_2080_4a80_a243_d37ec2ed7076.slice/crio-8c9524d350aebf32034509752f6b14b817feeec161bdf3e21a118a654a8e8494 WatchSource:0}: Error finding container 8c9524d350aebf32034509752f6b14b817feeec161bdf3e21a118a654a8e8494: Status 404 returned error can't find the container with id 8c9524d350aebf32034509752f6b14b817feeec161bdf3e21a118a654a8e8494
Nov 02 13:59:49.815014 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.814976    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" event={"ID":"e06878a8-2080-4a80-a243-d37ec2ed7076","Type":"ContainerStarted","Data":"72fe0e3de55b7fbd25af30a4a2da4f3948057cc503e3655b9a04b7512024da8f"}
Nov 02 13:59:49.815510 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.815026    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" event={"ID":"e06878a8-2080-4a80-a243-d37ec2ed7076","Type":"ContainerStarted","Data":"8c9524d350aebf32034509752f6b14b817feeec161bdf3e21a118a654a8e8494"}
Nov 02 13:59:49.816550 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.816528    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"
Nov 02 13:59:49.816683 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.816565    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx" event={"ID":"b9366740-9c2a-4df2-b1b1-e11f7beb2511","Type":"ContainerDied","Data":"41b1cea2fea7cb53455f80c184641e268529c39c98cb49da6c464d4600b818f7"}
Nov 02 13:59:49.816683 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.816608    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41b1cea2fea7cb53455f80c184641e268529c39c98cb49da6c464d4600b818f7"
Nov 02 13:59:49.859313 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.859264    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podStartSLOduration=1.859251282 podStartE2EDuration="1.859251282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:49.858408732 +0000 UTC m=+980.609713195" watchObservedRunningTime="2026-11-02 13:59:49.859251282 +0000 UTC m=+980.610555719"
Nov 02 13:59:49.950667 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:49.950641    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:50.007929 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.007883    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:50.060180 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.060147    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90beea4a-ca68-454f-95c9-d7576b8d8c8d-kube-api-access-qkwnt\" (UniqueName: \"kubernetes.io/projected/90beea4a-ca68-454f-95c9-d7576b8d8c8d-kube-api-access-qkwnt\") pod \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") "
Nov 02 13:59:50.060351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.060338    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-xks-ca\") pod \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") "
Nov 02 13:59:50.060478 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.060458    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-openshift-ca\") pod \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\" (UID: \"90beea4a-ca68-454f-95c9-d7576b8d8c8d\") "
Nov 02 13:59:50.060897 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.060860    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-openshift-ca" pod "90beea4a-ca68-454f-95c9-d7576b8d8c8d" (UID: "90beea4a-ca68-454f-95c9-d7576b8d8c8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:50.061101 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.060993    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:50.062542 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.062513    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90beea4a-ca68-454f-95c9-d7576b8d8c8d-kube-api-access-qkwnt" pod "90beea4a-ca68-454f-95c9-d7576b8d8c8d" (UID: "90beea4a-ca68-454f-95c9-d7576b8d8c8d"). InnerVolumeSpecName "kube-api-access-qkwnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:50.062985 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.062965    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-xks-ca" pod "90beea4a-ca68-454f-95c9-d7576b8d8c8d" (UID: "90beea4a-ca68-454f-95c9-d7576b8d8c8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:50.162262 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.162236    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90beea4a-ca68-454f-95c9-d7576b8d8c8d-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:50.162262 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.162261    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkwnt\" (UniqueName: \"kubernetes.io/projected/90beea4a-ca68-454f-95c9-d7576b8d8c8d-kube-api-access-qkwnt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:50.830712 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.830680    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"
Nov 02 13:59:50.830712 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.830692    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s" event={"ID":"90beea4a-ca68-454f-95c9-d7576b8d8c8d","Type":"ContainerDied","Data":"6c8041cab03f42a819f9db7f6b2ba4c8593eb1825581692d28eeffbe7f7aa9bf"}
Nov 02 13:59:50.831292 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:50.830755    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c8041cab03f42a819f9db7f6b2ba4c8593eb1825581692d28eeffbe7f7aa9bf"
Nov 02 13:59:52.994196 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:52.994164    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"]
Nov 02 13:59:52.996099 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:52.996068    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b9366740-9c2a-4df2-b1b1-e11f7beb2511" containerName="revoke-keys"
Nov 02 13:59:52.996099 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:52.996097    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9366740-9c2a-4df2-b1b1-e11f7beb2511" containerName="revoke-keys"
Nov 02 13:59:52.996278 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:52.996229    5034 cpu_manager.go:414] "containerMap: removing container" podUID="90beea4a-ca68-454f-95c9-d7576b8d8c8d" containerName="revoke-keys"
Nov 02 13:59:52.996278 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:52.996238    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90beea4a-ca68-454f-95c9-d7576b8d8c8d" containerName="revoke-keys"
Nov 02 13:59:52.996558 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:52.996542    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="90beea4a-ca68-454f-95c9-d7576b8d8c8d" containerName="revoke-keys"
Nov 02 13:59:52.996609 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:52.996563    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9366740-9c2a-4df2-b1b1-e11f7beb2511" containerName="revoke-keys"
Nov 02 13:59:53.006671 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.006646    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.018833 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.018788    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"]
Nov 02 13:59:53.045136 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.045106    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:53.045264 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.045167    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 13:59:53.049058 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.049040    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 13:59:53.049268 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.049251    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 13:59:53.106070 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.106041    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf295fdf-x8zkx\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.106209 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.106079    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4xs2\" (UniqueName: \"kubernetes.io/projected/b259e13e-1592-4c14-9914-c3c60a29bc3f-kube-api-access-q4xs2\") pod \"maas-api-revoke-sub-bf295fdf-x8zkx\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.106302 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.106274    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf295fdf-x8zkx\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.207310 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.207281    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf295fdf-x8zkx\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.207458 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.207430    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf295fdf-x8zkx\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.207458 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.207456    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xs2\" (UniqueName: \"kubernetes.io/projected/b259e13e-1592-4c14-9914-c3c60a29bc3f-kube-api-access-q4xs2\") pod \"maas-api-revoke-sub-bf295fdf-x8zkx\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.207988 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.207963    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf295fdf-x8zkx\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.209937 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.209912    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf295fdf-x8zkx\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.231612 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.231582    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xs2\" (UniqueName: \"kubernetes.io/projected/b259e13e-1592-4c14-9914-c3c60a29bc3f-kube-api-access-q4xs2\") pod \"maas-api-revoke-sub-bf295fdf-x8zkx\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.322881 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.322815    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:53.636052 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:53.636019    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb259e13e_1592_4c14_9914_c3c60a29bc3f.slice/crio-72963a696b6d3c7f05ac49595a7e0505b13aae49a1a04fab3442820aed8e5b75 WatchSource:0}: Error finding container 72963a696b6d3c7f05ac49595a7e0505b13aae49a1a04fab3442820aed8e5b75: Status 404 returned error can't find the container with id 72963a696b6d3c7f05ac49595a7e0505b13aae49a1a04fab3442820aed8e5b75
Nov 02 13:59:53.637821 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.637792    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"]
Nov 02 13:59:53.855542 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:53.855504    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx" event={"ID":"b259e13e-1592-4c14-9914-c3c60a29bc3f","Type":"ContainerStarted","Data":"72963a696b6d3c7f05ac49595a7e0505b13aae49a1a04fab3442820aed8e5b75"}
Nov 02 13:59:54.863947 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:54.863874    5034 generic.go:356] "Generic (PLEG): container finished" podID="b259e13e-1592-4c14-9914-c3c60a29bc3f" containerID="88b337407b57d3bfba6f4a1069973c9cd4dbf10fc255a5eac7cbff23bb4fb30d" exitCode=0
Nov 02 13:59:54.864282 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:54.863955    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx" event={"ID":"b259e13e-1592-4c14-9914-c3c60a29bc3f","Type":"ContainerDied","Data":"88b337407b57d3bfba6f4a1069973c9cd4dbf10fc255a5eac7cbff23bb4fb30d"}
Nov 02 13:59:56.063500 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.063478    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:56.246233 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.246204    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b259e13e-1592-4c14-9914-c3c60a29bc3f-kube-api-access-q4xs2\" (UniqueName: \"kubernetes.io/projected/b259e13e-1592-4c14-9914-c3c60a29bc3f-kube-api-access-q4xs2\") pod \"b259e13e-1592-4c14-9914-c3c60a29bc3f\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") "
Nov 02 13:59:56.246388 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.246351    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-openshift-ca\") pod \"b259e13e-1592-4c14-9914-c3c60a29bc3f\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") "
Nov 02 13:59:56.246512 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.246499    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-xks-ca\") pod \"b259e13e-1592-4c14-9914-c3c60a29bc3f\" (UID: \"b259e13e-1592-4c14-9914-c3c60a29bc3f\") "
Nov 02 13:59:56.246695 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.246673    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-openshift-ca" pod "b259e13e-1592-4c14-9914-c3c60a29bc3f" (UID: "b259e13e-1592-4c14-9914-c3c60a29bc3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:56.246966 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.246945    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:56.248328 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.248304    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b259e13e-1592-4c14-9914-c3c60a29bc3f-kube-api-access-q4xs2" pod "b259e13e-1592-4c14-9914-c3c60a29bc3f" (UID: "b259e13e-1592-4c14-9914-c3c60a29bc3f"). InnerVolumeSpecName "kube-api-access-q4xs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:56.248429 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.248418    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-xks-ca" pod "b259e13e-1592-4c14-9914-c3c60a29bc3f" (UID: "b259e13e-1592-4c14-9914-c3c60a29bc3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:56.348036 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.348007    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b259e13e-1592-4c14-9914-c3c60a29bc3f-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:56.348036 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.348035    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4xs2\" (UniqueName: \"kubernetes.io/projected/b259e13e-1592-4c14-9914-c3c60a29bc3f-kube-api-access-q4xs2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 13:59:56.882535 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.882506    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"
Nov 02 13:59:56.882712 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.882536    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx" event={"ID":"b259e13e-1592-4c14-9914-c3c60a29bc3f","Type":"ContainerDied","Data":"72963a696b6d3c7f05ac49595a7e0505b13aae49a1a04fab3442820aed8e5b75"}
Nov 02 13:59:56.882712 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:56.882615    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72963a696b6d3c7f05ac49595a7e0505b13aae49a1a04fab3442820aed8e5b75"
Nov 02 13:59:58.166239 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.166200    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"]
Nov 02 13:59:58.167136 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.167116    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b259e13e-1592-4c14-9914-c3c60a29bc3f" containerName="revoke-keys"
Nov 02 13:59:58.167192 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.167139    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b259e13e-1592-4c14-9914-c3c60a29bc3f" containerName="revoke-keys"
Nov 02 13:59:58.167311 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.167299    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b259e13e-1592-4c14-9914-c3c60a29bc3f" containerName="revoke-keys"
Nov 02 13:59:58.171085 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.171062    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.198217 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.198191    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"]
Nov 02 13:59:58.221591 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.221565    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"]
Nov 02 13:59:58.226336 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.226316    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.253772 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.253665    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"]
Nov 02 13:59:58.272841 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.272807    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd355c18-4dfd-45e3-848f-7d462d728762-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.273261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.272882    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.273261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.272942    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd355c18-4dfd-45e3-848f-7d462d728762-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.273261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.273031    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.273261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.273130    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.273261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.273166    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.273261 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.273191    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.273672 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.273277    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.273672 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.273320    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgmxx\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-kube-api-access-cgmxx\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.374907 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.374882    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375048 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.374919    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgmxx\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-kube-api-access-cgmxx\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375048 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.374943    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.375048 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375022    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd355c18-4dfd-45e3-848f-7d462d728762-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375238 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375086    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375238 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375148    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd355c18-4dfd-45e3-848f-7d462d728762-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375234    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375269    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.375351 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375325    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsjg6\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-kube-api-access-vsjg6\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.375466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375357    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.375466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375354    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375466 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375442    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.375553 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375481    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.375601 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375580    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375643 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375618    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375694 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375642    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.375694 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375668    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.375871 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375729    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.375871 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375771    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.376137 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.376114    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.376212 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.375882    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.376278 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.376228    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd355c18-4dfd-45e3-848f-7d462d728762-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.376417 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.376278    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.377983 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.377957    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd355c18-4dfd-45e3-848f-7d462d728762-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.378475 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.378452    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.400175 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.400151    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgmxx\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-kube-api-access-cgmxx\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.400768 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.400740    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.477226 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477167    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477326 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477224    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477326 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477286    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477477 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477461    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477544 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477514    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsjg6\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-kube-api-access-vsjg6\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477543    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477570    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477603 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477586    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-data\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477750 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477593    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477750 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477706    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.477913 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.477828    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-credential-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.478164 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.478138    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-socket\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.478239 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.478219    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-certs\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.478576 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.478549    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istiod-ca-cert\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.479703 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.479678    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-envoy\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.480307 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.480286    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-podinfo\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.488203 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.488181    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:58.502980 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.502957    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsjg6\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-kube-api-access-vsjg6\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.503121 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.503104    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-token\") pod \"e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") " pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.543851 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.543820    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 13:59:58.811576 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:58.811540    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd355c18_4dfd_45e3_848f_7d462d728762.slice/crio-6e895e70b4b8a5b86ffed36fedda1ea1e5a624036b3bfc40c28333969a215db1 WatchSource:0}: Error finding container 6e895e70b4b8a5b86ffed36fedda1ea1e5a624036b3bfc40c28333969a215db1: Status 404 returned error can't find the container with id 6e895e70b4b8a5b86ffed36fedda1ea1e5a624036b3bfc40c28333969a215db1
Nov 02 13:59:58.813367 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.813271    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"]
Nov 02 13:59:58.814498 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.814458    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:58.814590 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.814543    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:58.814638 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.814587    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:58.882738 ip-10-0-139-26 kubenswrapper[5034]: W1102 13:59:58.882708    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2ff075_0f93_4dc6_86f4_5b0e2a50b4e4.slice/crio-37f9eb6d25e3ff257c8264e16c2492b16dd437d7e0e4fcd51877ae8275e9781d WatchSource:0}: Error finding container 37f9eb6d25e3ff257c8264e16c2492b16dd437d7e0e4fcd51877ae8275e9781d: Status 404 returned error can't find the container with id 37f9eb6d25e3ff257c8264e16c2492b16dd437d7e0e4fcd51877ae8275e9781d
Nov 02 13:59:58.885451 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.885427    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"]
Nov 02 13:59:58.885838 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.885796    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:58.885935 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.885893    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:58.885972 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.885945    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 13:59:58.903180 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.903106    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" event={"ID":"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4","Type":"ContainerStarted","Data":"37f9eb6d25e3ff257c8264e16c2492b16dd437d7e0e4fcd51877ae8275e9781d"}
Nov 02 13:59:58.905610 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.905581    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" event={"ID":"fd355c18-4dfd-45e3-848f-7d462d728762","Type":"ContainerStarted","Data":"d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959"}
Nov 02 13:59:58.905734 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.905635    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" event={"ID":"fd355c18-4dfd-45e3-848f-7d462d728762","Type":"ContainerStarted","Data":"6e895e70b4b8a5b86ffed36fedda1ea1e5a624036b3bfc40c28333969a215db1"}
Nov 02 13:59:58.963859 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.963773    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podStartSLOduration=0.963753825 podStartE2EDuration="963.753825ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:58.962344247 +0000 UTC m=+989.713648686" watchObservedRunningTime="2026-11-02 13:59:58.963753825 +0000 UTC m=+989.715058263"
Nov 02 13:59:58.998730 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:58.998708    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:59.022336 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:59.022057    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 13:59:59.493229 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:59.493190    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 13:59:59.493996 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:59.493968    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 13:59:59.494183 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:59.494160    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 13:59:59.944459 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:59.944368    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" event={"ID":"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4","Type":"ContainerStarted","Data":"337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612"}
Nov 02 13:59:59.987802 ip-10-0-139-26 kubenswrapper[5034]: I1102 13:59:59.979311    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:00:00.029850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:00.026871    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podStartSLOduration=2.026833365 podStartE2EDuration="2.026833365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:00.012011455 +0000 UTC m=+990.763315894" watchObservedRunningTime="2026-11-02 14:00:00.026833365 +0000 UTC m=+990.778137804"
Nov 02 14:00:00.496802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:00.491131    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:00.496802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:00.491192    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:00.696706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:00.580957    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 14:00:00.696706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:00.650091    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:00.696706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:00.650156    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:00.696706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:00.650410    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"]
Nov 02 14:00:00.961221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:00.961141    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"]
Nov 02 14:00:00.997849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:00.986922    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:01.030834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.016911    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:00:01.030834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.017613    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:00:01.030834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.017706    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:00:01.032798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.032746    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20200d2b-c009-4cee-86df-d8c05b8ca507-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45\" (UID: \"20200d2b-c009-4cee-86df-d8c05b8ca507\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:01.032955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.032933    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlpc\" (UniqueName: \"kubernetes.io/projected/20200d2b-c009-4cee-86df-d8c05b8ca507-kube-api-access-rmlpc\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45\" (UID: \"20200d2b-c009-4cee-86df-d8c05b8ca507\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:01.033416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.033383    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"]
Nov 02 14:00:01.040843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.040815    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:01.064511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.064475    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"]
Nov 02 14:00:01.078812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.077704    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:01.080207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.080180    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"]
Nov 02 14:00:01.080470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.080454    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:01.110801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.109956    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"]
Nov 02 14:00:01.139840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.139805    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlpc\" (UniqueName: \"kubernetes.io/projected/20200d2b-c009-4cee-86df-d8c05b8ca507-kube-api-access-rmlpc\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45\" (UID: \"20200d2b-c009-4cee-86df-d8c05b8ca507\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:01.140229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.140209    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20200d2b-c009-4cee-86df-d8c05b8ca507-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45\" (UID: \"20200d2b-c009-4cee-86df-d8c05b8ca507\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:01.156274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.156241    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20200d2b-c009-4cee-86df-d8c05b8ca507-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45\" (UID: \"20200d2b-c009-4cee-86df-d8c05b8ca507\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:01.171681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.171620    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:01.209527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.209497    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"]
Nov 02 14:00:01.243833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.243206    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a886243d-85ee-4423-878e-4cf8a02772b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh\" (UID: \"a886243d-85ee-4423-878e-4cf8a02772b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:01.251824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.249754    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlpc\" (UniqueName: \"kubernetes.io/projected/20200d2b-c009-4cee-86df-d8c05b8ca507-kube-api-access-rmlpc\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45\" (UID: \"20200d2b-c009-4cee-86df-d8c05b8ca507\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:01.251824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.243294    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0b86e4-42b4-4c36-9691-78b40bffd94c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v\" (UID: \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:01.251824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.250399    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5a92e1-a601-4634-93a5-3a7cbbb15168-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw\" (UID: \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:01.251824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.250480    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqrf4\" (UniqueName: \"kubernetes.io/projected/ec5a92e1-a601-4634-93a5-3a7cbbb15168-kube-api-access-vqrf4\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw\" (UID: \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:01.251824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.250597    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h58gz\" (UniqueName: \"kubernetes.io/projected/8c0b86e4-42b4-4c36-9691-78b40bffd94c-kube-api-access-h58gz\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v\" (UID: \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:01.251824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.250824    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r779b\" (UniqueName: \"kubernetes.io/projected/a886243d-85ee-4423-878e-4cf8a02772b3-kube-api-access-r779b\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh\" (UID: \"a886243d-85ee-4423-878e-4cf8a02772b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:01.273269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.273226    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"]
Nov 02 14:00:01.334942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.334891    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:01.384804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.373715    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0b86e4-42b4-4c36-9691-78b40bffd94c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v\" (UID: \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:01.384804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.373891    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5a92e1-a601-4634-93a5-3a7cbbb15168-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw\" (UID: \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:01.384804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.373949    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqrf4\" (UniqueName: \"kubernetes.io/projected/ec5a92e1-a601-4634-93a5-3a7cbbb15168-kube-api-access-vqrf4\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw\" (UID: \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:01.384804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.374021    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h58gz\" (UniqueName: \"kubernetes.io/projected/8c0b86e4-42b4-4c36-9691-78b40bffd94c-kube-api-access-h58gz\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v\" (UID: \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:01.384804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.374167    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r779b\" (UniqueName: \"kubernetes.io/projected/a886243d-85ee-4423-878e-4cf8a02772b3-kube-api-access-r779b\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh\" (UID: \"a886243d-85ee-4423-878e-4cf8a02772b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:01.384804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.374217    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a886243d-85ee-4423-878e-4cf8a02772b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh\" (UID: \"a886243d-85ee-4423-878e-4cf8a02772b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:01.384804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.375095    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a886243d-85ee-4423-878e-4cf8a02772b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh\" (UID: \"a886243d-85ee-4423-878e-4cf8a02772b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:01.384804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.375705    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0b86e4-42b4-4c36-9691-78b40bffd94c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v\" (UID: \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:01.384804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.376380    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5a92e1-a601-4634-93a5-3a7cbbb15168-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw\" (UID: \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:01.433682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.433644    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqrf4\" (UniqueName: \"kubernetes.io/projected/ec5a92e1-a601-4634-93a5-3a7cbbb15168-kube-api-access-vqrf4\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw\" (UID: \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:01.446802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.434516    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:01.446802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.441639    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h58gz\" (UniqueName: \"kubernetes.io/projected/8c0b86e4-42b4-4c36-9691-78b40bffd94c-kube-api-access-h58gz\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v\" (UID: \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:01.446802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.442248    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:01.453827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.449571    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r779b\" (UniqueName: \"kubernetes.io/projected/a886243d-85ee-4423-878e-4cf8a02772b3-kube-api-access-r779b\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh\" (UID: \"a886243d-85ee-4423-878e-4cf8a02772b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:01.504103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.504005    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:01.505333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.505294    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:01.539799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.524078    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:01.556811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.553226    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:01.556811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.554122    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:01.556811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.554285    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:01.626278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.610057    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:01.694809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.688896    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:01.860901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.859914    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:01.970375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:01.970307    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:00:02.034511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.034466    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:02.173709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.173667    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:02.181857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.176931    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:02.195902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.183720    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:02.196406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.183750    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:00:02.196808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.183771    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:02.378864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.343994    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:02.378864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.344284    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:00:02.487804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.447440    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:00:02.487804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.472903    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:00:02.753877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.739209    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:02.753877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.739281    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:02.753877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.739846    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:02.753877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.739869    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:00:02.753877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.739879    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:02.753877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.740249    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:00:02.753877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.746370    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Nov 02 14:00:02.981804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.981183    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:02.981804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:02.981475    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:00:03.058863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.053413    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:03.058863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.054443    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:00:03.085831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.073951    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:03.085831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.076936    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:00:03.179932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.179882    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:03.180893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.180872    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:00:03.466590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.466496    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:03.466917 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.466892    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:03.467089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.467069    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:00:03.467216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.467201    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:03.467408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.467391    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:00:03.556309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.556272    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:00:03.575002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.574967    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:00:03.575386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.575357    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:00:03.581399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.581363    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:03.594927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.594872    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:03.781080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.743980    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:00:03.826033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.825994    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:03.826350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.826302    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:03.950503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:03.950463    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.017677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.017627    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.057809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.057690    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:04.119224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.119167    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.194807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.190932    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:00:04.350263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.328964    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:04.350263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.329065    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.487462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.487411    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.642497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.532723    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.676883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.532761    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:04.707839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.587881    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.707839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.681160    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:04.707839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.681205    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.749204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.749158    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:04.929380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.904300    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:04.929380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.904359    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:04.929380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.909414    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:04.929380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:04.909472    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:05.096788 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:05.096003    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:05.155272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:05.155227    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:05.230297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:05.230196    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:05.235218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:05.205893    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:05.280359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:05.280310    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:05.578426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:05.578380    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:05.578707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:05.578681    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:05.629805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:05.628926    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:05.629805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:05.629011    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:06.244926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.232947    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:06.244926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.238036    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:06.251589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.251516    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:06.339307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.339070    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp" podUID="d345d401-090c-4d81-a5fc-fbf5ca938209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded"
Nov 02 14:00:06.372856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.363497    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:06.401807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.376723    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp" podUID="e71f9623-d37b-4294-a11b-a408ecc42eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 02 14:00:06.521862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.470080    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:06.587802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.495240    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:00:06.704887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.658881    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:06.704887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.689949    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:06.704887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.663037    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:06.704887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.690082    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:06.704887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:06.690603    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:07.146183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.143934    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:07.146183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.144008    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:07.304670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.262809    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:07.359898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.338865    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:07.372033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.359865    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp" podUID="d345d401-090c-4d81-a5fc-fbf5ca938209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:07.372033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.363918    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:07.434154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.432589    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:07.458846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.448249    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:00:07.458846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.449711    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:00:07.458846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.450955    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:07.458846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.450994    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:07.529844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.527915    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded"
Nov 02 14:00:07.540594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.540554    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:07.626867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.545594    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:07.626867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.545644    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:07.626867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.545738    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:07.626867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.545758    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:07.626867 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:07.552246    5034 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.016s"
Nov 02 14:00:07.626867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.569863    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:07.703144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.698917    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:00:07.703144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.699374    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:07.744995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.744962    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 14:00:07.745438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.745415    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"]
Nov 02 14:00:07.758022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.757987    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" event={"ID":"8c0b86e4-42b4-4c36-9691-78b40bffd94c","Type":"ContainerStarted","Data":"9837afb4f71923fa91edb367475b9c5730aa35f4226735048ce73ac8eb30878c"}
Nov 02 14:00:07.758301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.758286    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"]
Nov 02 14:00:07.758864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.758247    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783464    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783503    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp"
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783529    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" event={"ID":"ec5a92e1-a601-4634-93a5-3a7cbbb15168","Type":"ContainerStarted","Data":"2153f2ab1e4c5cfa44fa18ebdc66b4462ba40832b89cb6188c4dd761f7a6f0bb"}
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783566    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"]
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783603    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"]
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783629    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" event={"ID":"20200d2b-c009-4cee-86df-d8c05b8ca507","Type":"ContainerStarted","Data":"48e7226a14569053cb9f0efc0b2520749d68fbd1782df0168cea4a810294559f"}
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783675    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783714    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783748    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783801    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783835    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783870    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:00:07.786740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.783901    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:00:07.800886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.788472    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:07.800886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.793291    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:00:07.851920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.848210    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"]
Nov 02 14:00:07.861942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.861896    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"]
Nov 02 14:00:07.862179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.861950    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"]
Nov 02 14:00:07.862179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.861982    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"]
Nov 02 14:00:07.862179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.862040    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:00:07.862964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.862932    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:07.889263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.887637    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:07.912805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.895993    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"]
Nov 02 14:00:07.914254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.914215    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.042157    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.042207    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c53b23c-8143-491f-8704-03fbf789f473-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.042360    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.042433    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.042566    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.042608    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2851278-2635-49f8-a671-02124f264846-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:07.936851    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"]
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.042657    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.042691    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh" event={"ID":"a886243d-85ee-4423-878e-4cf8a02772b3","Type":"ContainerStarted","Data":"e060394456dd151aa35ee71f4a0aa862d6dda1a157b9ebc4e4c5cbf94b9b6f2b"}
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.042926    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.043209    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.043278    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2851278-2635-49f8-a671-02124f264846-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.043808    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg68n\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-kube-api-access-kg68n\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.049838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.044749    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.060804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.055072    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.063243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.063196    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.063243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.063245    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.063470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.063272    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-728xr\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-kube-api-access-728xr\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.063470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.063370    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c53b23c-8143-491f-8704-03fbf789f473-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.165394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.164879    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.165394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.164950    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.165394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.164979    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c53b23c-8143-491f-8704-03fbf789f473-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.165394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.165012    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa9b9d92-9qw5w\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:08.165394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.165049    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.165394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.165083    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.165394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.165135    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.165158    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2851278-2635-49f8-a671-02124f264846-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.165987    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.166141    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.166233    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa9b9d92-9qw5w\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.166264    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.166301    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2851278-2635-49f8-a671-02124f264846-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.166345    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg68n\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-kube-api-access-kg68n\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.166385    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.166417    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67rxn\" (UniqueName: \"kubernetes.io/projected/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-kube-api-access-67rxn\") pod \"maas-api-revoke-sub-fa9b9d92-9qw5w\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.166447    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.166863    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.175878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.168181    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.175878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.168624    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.238081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.236508    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c53b23c-8143-491f-8704-03fbf789f473-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.238081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.236824    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.238081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.237084    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.238081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.237375    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.256117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.253045    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.256117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.253676    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.256117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.253925    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.256117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.255386    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.257855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.256615    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-728xr\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-kube-api-access-728xr\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.258860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.258834    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c53b23c-8143-491f-8704-03fbf789f473-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.265494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.264964    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2851278-2635-49f8-a671-02124f264846-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.326717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.326683    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.353429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.348872    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.359084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.359053    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c53b23c-8143-491f-8704-03fbf789f473-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.361302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.361266    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa9b9d92-9qw5w\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:08.361452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.361370    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67rxn\" (UniqueName: \"kubernetes.io/projected/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-kube-api-access-67rxn\") pod \"maas-api-revoke-sub-fa9b9d92-9qw5w\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:08.361558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.361541    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa9b9d92-9qw5w\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:08.362618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.362586    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa9b9d92-9qw5w\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:08.363557 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.363529    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.363807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.363762    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.364569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.364540    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg68n\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-kube-api-access-kg68n\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.365642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.365524    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-728xr\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-kube-api-access-728xr\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.367869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.366402    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2851278-2635-49f8-a671-02124f264846-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.456459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.456419    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:08.491289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.491244    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:08.491586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.491547    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:08.553132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.553084    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:08.553902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.553852    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:08.555801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.555031    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:08.621817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.593851    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa9b9d92-9qw5w\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:08.650090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.650051    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67rxn\" (UniqueName: \"kubernetes.io/projected/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-kube-api-access-67rxn\") pod \"maas-api-revoke-sub-fa9b9d92-9qw5w\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:08.654425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:08.654181    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:09.456449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.453042    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh" podStartSLOduration=9.453018002 podStartE2EDuration="9.453018002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:09.391209386 +0000 UTC m=+1000.142513825" watchObservedRunningTime="2026-11-02 14:00:09.453018002 +0000 UTC m=+1000.204322440"
Nov 02 14:00:09.559268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.489554    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:09.559268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.489616    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:09.559268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.540417    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"]
Nov 02 14:00:09.559268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.540702    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"]
Nov 02 14:00:09.559268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.552695    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:09.559268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.552747    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:09.607472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.595491    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:00:09.607472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.595622    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:00:09.607472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.595688    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:00:09.615236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.615175    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:00:09.622945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.622896    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:00:09.623275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.623246    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:00:09.650503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:09.650467    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"]
Nov 02 14:00:09.687618 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:00:09.684018    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8643d5d_ca1c_4c95_9281_5f1e48b77ddd.slice/crio-bd6a1491b7d6d460baf775f4944abfb1d62e5b3ff3210031b64d1d14aa098953 WatchSource:0}: Error finding container bd6a1491b7d6d460baf775f4944abfb1d62e5b3ff3210031b64d1d14aa098953: Status 404 returned error can't find the container with id bd6a1491b7d6d460baf775f4944abfb1d62e5b3ff3210031b64d1d14aa098953
Nov 02 14:00:10.266821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.265827    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" event={"ID":"8c0b86e4-42b4-4c36-9691-78b40bffd94c","Type":"ContainerDied","Data":"069b2fefd9d03195351bc246470cdd382e8915048ea9cab34abc712996f53efd"}
Nov 02 14:00:10.266821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.266248    5034 scope.go:122] "RemoveContainer" containerID="069b2fefd9d03195351bc246470cdd382e8915048ea9cab34abc712996f53efd"
Nov 02 14:00:10.266821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.266220    5034 generic.go:356] "Generic (PLEG): container finished" podID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerID="069b2fefd9d03195351bc246470cdd382e8915048ea9cab34abc712996f53efd" exitCode=60
Nov 02 14:00:10.282758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.280368    5034 generic.go:356] "Generic (PLEG): container finished" podID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerID="8a4d03bcaf6faaea8d32a5afb1587c4337b2d4957b70d78264856941c5304fc4" exitCode=60
Nov 02 14:00:10.282758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.280495    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" event={"ID":"ec5a92e1-a601-4634-93a5-3a7cbbb15168","Type":"ContainerDied","Data":"8a4d03bcaf6faaea8d32a5afb1587c4337b2d4957b70d78264856941c5304fc4"}
Nov 02 14:00:10.282758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.280955    5034 scope.go:122] "RemoveContainer" containerID="8a4d03bcaf6faaea8d32a5afb1587c4337b2d4957b70d78264856941c5304fc4"
Nov 02 14:00:10.287991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.286987    5034 generic.go:356] "Generic (PLEG): container finished" podID="a886243d-85ee-4423-878e-4cf8a02772b3" containerID="e8eed56c02bfd276fe19f4811754f5131403e6c8abd9f87e6cf611c84c37de19" exitCode=60
Nov 02 14:00:10.287991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.287151    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh" event={"ID":"a886243d-85ee-4423-878e-4cf8a02772b3","Type":"ContainerDied","Data":"e8eed56c02bfd276fe19f4811754f5131403e6c8abd9f87e6cf611c84c37de19"}
Nov 02 14:00:10.287991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.287566    5034 scope.go:122] "RemoveContainer" containerID="e8eed56c02bfd276fe19f4811754f5131403e6c8abd9f87e6cf611c84c37de19"
Nov 02 14:00:10.296114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.295588    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w" event={"ID":"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd","Type":"ContainerStarted","Data":"bd6a1491b7d6d460baf775f4944abfb1d62e5b3ff3210031b64d1d14aa098953"}
Nov 02 14:00:10.308531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.306897    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" event={"ID":"d2851278-2635-49f8-a671-02124f264846","Type":"ContainerStarted","Data":"269e4cc45cde1310267ab38cbc93d52e1c6e52615c75a107bc7a0fcdcbd81946"}
Nov 02 14:00:10.308531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.306960    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" event={"ID":"d2851278-2635-49f8-a671-02124f264846","Type":"ContainerStarted","Data":"d825725940b4e21b2d62579a84e7ed783c4ad57daca951a2eff3bd66ed376c9f"}
Nov 02 14:00:10.321304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.321272    5034 generic.go:356] "Generic (PLEG): container finished" podID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerID="4075dcda5ce150031a50d81be83ff9e7cd41960859de29cfa886c96bf53b7373" exitCode=60
Nov 02 14:00:10.321638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.321613    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" event={"ID":"20200d2b-c009-4cee-86df-d8c05b8ca507","Type":"ContainerDied","Data":"4075dcda5ce150031a50d81be83ff9e7cd41960859de29cfa886c96bf53b7373"}
Nov 02 14:00:10.322618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.322598    5034 scope.go:122] "RemoveContainer" containerID="4075dcda5ce150031a50d81be83ff9e7cd41960859de29cfa886c96bf53b7373"
Nov 02 14:00:10.328288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.328260    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" event={"ID":"5c53b23c-8143-491f-8704-03fbf789f473","Type":"ContainerStarted","Data":"957ffbfa1b3a545540ce027bcf84727d1a4b748f09f3f575ca52c21b8857a36c"}
Nov 02 14:00:10.328457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.328441    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" event={"ID":"5c53b23c-8143-491f-8704-03fbf789f473","Type":"ContainerStarted","Data":"aa6616b9441e6aaa3b548c2c842f70d8e21f63d7120941c6e81391bdb16bed7d"}
Nov 02 14:00:10.463967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.463870    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:10.521175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.490215    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 02 14:00:10.521175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.490271    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 02 14:00:10.555572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.555510    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" podStartSLOduration=6.555489493 podStartE2EDuration="6.555489493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:10.489602779 +0000 UTC m=+1001.240907219" watchObservedRunningTime="2026-11-02 14:00:10.555489493 +0000 UTC m=+1001.306793933"
Nov 02 14:00:10.557138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.557110    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:10.563799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.557631    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 02 14:00:10.563799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.557680    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 02 14:00:10.567798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:10.564615    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" podStartSLOduration=6.56459792 podStartE2EDuration="6.56459792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:10.559247363 +0000 UTC m=+1001.310551803" watchObservedRunningTime="2026-11-02 14:00:10.56459792 +0000 UTC m=+1001.315902359"
Nov 02 14:00:11.378547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.378450    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh" event={"ID":"a886243d-85ee-4423-878e-4cf8a02772b3","Type":"ContainerStarted","Data":"3b2266365f0e6f1f2a38eca135ba6a389bd76d9d61cad39c68eb7a3dd50eb9d1"}
Nov 02 14:00:11.444013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.443482    5034 generic.go:356] "Generic (PLEG): container finished" podID="e8643d5d-ca1c-4c95-9281-5f1e48b77ddd" containerID="1a713596fd2e2cc9a699e6a5521ec23d34a348d6040e87ab7a2705339f08854d" exitCode=0
Nov 02 14:00:11.456803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.453103    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w" event={"ID":"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd","Type":"ContainerDied","Data":"1a713596fd2e2cc9a699e6a5521ec23d34a348d6040e87ab7a2705339f08854d"}
Nov 02 14:00:11.481476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.481438    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" event={"ID":"20200d2b-c009-4cee-86df-d8c05b8ca507","Type":"ContainerStarted","Data":"9ecc62c5941343fd6afa335777ffdc2bdbbf0dd0c1f7f24ea14e15778bb73451"}
Nov 02 14:00:11.482217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.458692    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 14:00:11.482376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.482349    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 14:00:11.529192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.529152    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" event={"ID":"8c0b86e4-42b4-4c36-9691-78b40bffd94c","Type":"ContainerStarted","Data":"d51d3437a353448d0da3dac913b7594f847d3a77211f9646e5053ecbc82d8931"}
Nov 02 14:00:11.540267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.540230    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" event={"ID":"ec5a92e1-a601-4634-93a5-3a7cbbb15168","Type":"ContainerStarted","Data":"629f35db5d2d691f7b8caa99b157686ee112f4b81e12381dcc36763f6166b253"}
Nov 02 14:00:11.561598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.561563    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 14:00:11.561906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.561875    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 14:00:11.563822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.563755    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" podStartSLOduration=11.563736126 podStartE2EDuration="11.563736126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:11.526096542 +0000 UTC m=+1002.277400981" watchObservedRunningTime="2026-11-02 14:00:11.563736126 +0000 UTC m=+1002.315040564"
Nov 02 14:00:11.692747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:11.692674    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" podStartSLOduration=11.692651527 podStartE2EDuration="11.692651527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:11.64421538 +0000 UTC m=+1002.395519845" watchObservedRunningTime="2026-11-02 14:00:11.692651527 +0000 UTC m=+1002.443955968"
Nov 02 14:00:12.480813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.460186    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 14:00:12.480813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.460246    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 14:00:12.508810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.496892    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:12.508810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.496960    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:12.564520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.564066    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 14:00:12.564520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.564124    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 14:00:12.584024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.582865    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:12.584024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.582929    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:12.668017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.667977    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:12.701076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.701031    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:12.743633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.730460    5034 generic.go:356] "Generic (PLEG): container finished" podID="a886243d-85ee-4423-878e-4cf8a02772b3" containerID="3b2266365f0e6f1f2a38eca135ba6a389bd76d9d61cad39c68eb7a3dd50eb9d1" exitCode=60
Nov 02 14:00:12.743633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.730694    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh" event={"ID":"a886243d-85ee-4423-878e-4cf8a02772b3","Type":"ContainerDied","Data":"3b2266365f0e6f1f2a38eca135ba6a389bd76d9d61cad39c68eb7a3dd50eb9d1"}
Nov 02 14:00:12.743633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.730770    5034 scope.go:122] "RemoveContainer" containerID="e8eed56c02bfd276fe19f4811754f5131403e6c8abd9f87e6cf611c84c37de19"
Nov 02 14:00:12.743633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.731350    5034 scope.go:122] "RemoveContainer" containerID="3b2266365f0e6f1f2a38eca135ba6a389bd76d9d61cad39c68eb7a3dd50eb9d1"
Nov 02 14:00:12.743633 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:12.731765    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh_odh-ai-gateway-infra(a886243d-85ee-4423-878e-4cf8a02772b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh" podUID="a886243d-85ee-4423-878e-4cf8a02772b3"
Nov 02 14:00:12.779828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.778327    5034 generic.go:356] "Generic (PLEG): container finished" podID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerID="9ecc62c5941343fd6afa335777ffdc2bdbbf0dd0c1f7f24ea14e15778bb73451" exitCode=60
Nov 02 14:00:12.779828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.778514    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" event={"ID":"20200d2b-c009-4cee-86df-d8c05b8ca507","Type":"ContainerDied","Data":"9ecc62c5941343fd6afa335777ffdc2bdbbf0dd0c1f7f24ea14e15778bb73451"}
Nov 02 14:00:12.779828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.779083    5034 scope.go:122] "RemoveContainer" containerID="9ecc62c5941343fd6afa335777ffdc2bdbbf0dd0c1f7f24ea14e15778bb73451"
Nov 02 14:00:12.779828 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:12.779528    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45_odh-ai-gateway-infra(20200d2b-c009-4cee-86df-d8c05b8ca507)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507"
Nov 02 14:00:12.799819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.786799    5034 generic.go:356] "Generic (PLEG): container finished" podID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerID="d51d3437a353448d0da3dac913b7594f847d3a77211f9646e5053ecbc82d8931" exitCode=60
Nov 02 14:00:12.807271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.803919    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" event={"ID":"8c0b86e4-42b4-4c36-9691-78b40bffd94c","Type":"ContainerDied","Data":"d51d3437a353448d0da3dac913b7594f847d3a77211f9646e5053ecbc82d8931"}
Nov 02 14:00:12.807271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.804492    5034 scope.go:122] "RemoveContainer" containerID="d51d3437a353448d0da3dac913b7594f847d3a77211f9646e5053ecbc82d8931"
Nov 02 14:00:12.807271 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:12.804941    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v_odh-ai-gateway-infra(8c0b86e4-42b4-4c36-9691-78b40bffd94c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c"
Nov 02 14:00:12.813838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.808722    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" podStartSLOduration=12.808696637 podStartE2EDuration="12.808696637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:11.727921587 +0000 UTC m=+1002.479226031" watchObservedRunningTime="2026-11-02 14:00:12.808696637 +0000 UTC m=+1003.560001075"
Nov 02 14:00:12.860365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.860322    5034 scope.go:122] "RemoveContainer" containerID="4075dcda5ce150031a50d81be83ff9e7cd41960859de29cfa886c96bf53b7373"
Nov 02 14:00:12.861260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.861234    5034 generic.go:356] "Generic (PLEG): container finished" podID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerID="629f35db5d2d691f7b8caa99b157686ee112f4b81e12381dcc36763f6166b253" exitCode=60
Nov 02 14:00:12.862460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.862442    5034 scope.go:122] "RemoveContainer" containerID="629f35db5d2d691f7b8caa99b157686ee112f4b81e12381dcc36763f6166b253"
Nov 02 14:00:12.863156 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:12.863122    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw_odh-ai-gateway-infra(ec5a92e1-a601-4634-93a5-3a7cbbb15168)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168"
Nov 02 14:00:12.863328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:12.863307    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" event={"ID":"ec5a92e1-a601-4634-93a5-3a7cbbb15168","Type":"ContainerDied","Data":"629f35db5d2d691f7b8caa99b157686ee112f4b81e12381dcc36763f6166b253"}
Nov 02 14:00:13.181848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.114104    5034 scope.go:122] "RemoveContainer" containerID="069b2fefd9d03195351bc246470cdd382e8915048ea9cab34abc712996f53efd"
Nov 02 14:00:13.675599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.675507    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 14:00:13.683940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.683893    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 14:00:13.776359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.764316    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:13.776359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.764365    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:13.795373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.795313    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:13.823986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.823935    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:13.824265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.824233    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:13.882815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.881965    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:13.892881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.891471    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:14.003867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:13.973464    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Nov 02 14:00:14.195280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.195236    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 14:00:14.250151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.231155    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 14:00:14.250151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.219515    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:14.250151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.204378    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:14.324969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.311373    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:14.368539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.368492    5034 scope.go:122] "RemoveContainer" containerID="8a4d03bcaf6faaea8d32a5afb1587c4337b2d4957b70d78264856941c5304fc4"
Nov 02 14:00:14.407845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.397650    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-openshift-ca\") pod \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") "
Nov 02 14:00:14.407845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.397718    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-kube-api-access-67rxn\" (UniqueName: \"kubernetes.io/projected/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-kube-api-access-67rxn\") pod \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") "
Nov 02 14:00:14.407845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.397903    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-xks-ca\") pod \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\" (UID: \"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd\") "
Nov 02 14:00:14.409974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.409950    5034 scope.go:122] "RemoveContainer" containerID="9ecc62c5941343fd6afa335777ffdc2bdbbf0dd0c1f7f24ea14e15778bb73451"
Nov 02 14:00:14.429327 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:14.429277    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45_odh-ai-gateway-infra(20200d2b-c009-4cee-86df-d8c05b8ca507)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507"
Nov 02 14:00:14.431053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.430976    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-openshift-ca" pod "e8643d5d-ca1c-4c95-9281-5f1e48b77ddd" (UID: "e8643d5d-ca1c-4c95-9281-5f1e48b77ddd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:14.567246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.567205    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:14.654439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.654333    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:14.717938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.717887    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:14.897587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.897539    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:14.898749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.898712    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:14.919986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.919879    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:14.920301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.920271    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:14.948677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.948638    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 14:00:14.949816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.949740    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 14:00:14.949929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.949824    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 14:00:14.959849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.959818    5034 scope.go:122] "RemoveContainer" containerID="629f35db5d2d691f7b8caa99b157686ee112f4b81e12381dcc36763f6166b253"
Nov 02 14:00:14.960090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.960070    5034 scope.go:122] "RemoveContainer" containerID="d51d3437a353448d0da3dac913b7594f847d3a77211f9646e5053ecbc82d8931"
Nov 02 14:00:15.054800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:14.958384    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 14:00:15.054800 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:15.053926    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v_odh-ai-gateway-infra(8c0b86e4-42b4-4c36-9691-78b40bffd94c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c"
Nov 02 14:00:15.054800 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:15.054188    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw_odh-ai-gateway-infra(ec5a92e1-a601-4634-93a5-3a7cbbb15168)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168"
Nov 02 14:00:15.054800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.054623    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-kube-api-access-67rxn" pod "e8643d5d-ca1c-4c95-9281-5f1e48b77ddd" (UID: "e8643d5d-ca1c-4c95-9281-5f1e48b77ddd"). InnerVolumeSpecName "kube-api-access-67rxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:15.066877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.055220    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:15.066877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.057538    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-xks-ca" pod "e8643d5d-ca1c-4c95-9281-5f1e48b77ddd" (UID: "e8643d5d-ca1c-4c95-9281-5f1e48b77ddd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:15.088241 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:15.088197    5034 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.654s"
Nov 02 14:00:15.088663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.088642    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 14:00:15.089066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.089033    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 14:00:15.089291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.089277    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:00:15.089543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.089529    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:00:15.089738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.089725    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 14:00:15.114895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.114866    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 14:00:15.122804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.115185    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:00:15.122804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.115236    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:00:15.122804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.115271    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:00:15.122804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.115362    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:00:15.146494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.146453    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 14:00:15.146856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.146840    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 14:00:15.157878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.157833    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67rxn\" (UniqueName: \"kubernetes.io/projected/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-kube-api-access-67rxn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:15.157878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.157867    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:15.458051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.458017    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 14:00:15.458361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.458333    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 14:00:15.558346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.558313    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 14:00:15.558553 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.558523    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 14:00:15.940917 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.940880    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"
Nov 02 14:00:15.941821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.941757    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w" event={"ID":"e8643d5d-ca1c-4c95-9281-5f1e48b77ddd","Type":"ContainerDied","Data":"bd6a1491b7d6d460baf775f4944abfb1d62e5b3ff3210031b64d1d14aa098953"}
Nov 02 14:00:15.942138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:15.942119    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd6a1491b7d6d460baf775f4944abfb1d62e5b3ff3210031b64d1d14aa098953"
Nov 02 14:00:16.457266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:16.457227    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 14:00:16.457477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:16.457286    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 14:00:16.556956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:16.556925    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 14:00:16.557150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:16.556977    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 14:00:17.457460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:17.457422    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 02 14:00:17.457977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:17.457491    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 02 14:00:17.557473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:17.557437    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Nov 02 14:00:17.557684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:17.557502    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Nov 02 14:00:18.428791 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:18.428623    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:00:18.562604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:18.562570    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:18.563134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:18.563114    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:18.564011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:18.563988    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:00:18.763469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:18.763397    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:18.764135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:18.764100    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:18.767558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:18.767537    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:00:23.387487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:23.387454    5034 scope.go:122] "RemoveContainer" containerID="629f35db5d2d691f7b8caa99b157686ee112f4b81e12381dcc36763f6166b253"
Nov 02 14:00:23.388079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:23.388052    5034 scope.go:122] "RemoveContainer" containerID="d51d3437a353448d0da3dac913b7594f847d3a77211f9646e5053ecbc82d8931"
Nov 02 14:00:24.119801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:24.118287    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" event={"ID":"8c0b86e4-42b4-4c36-9691-78b40bffd94c","Type":"ContainerStarted","Data":"757e975cb2f79ce753c9674a8e9fa905636ff55d6ece222a03e45db4a0058606"}
Nov 02 14:00:24.150808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:24.144748    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" event={"ID":"ec5a92e1-a601-4634-93a5-3a7cbbb15168","Type":"ContainerStarted","Data":"5d4b190f0ad5af39b702d7b48291072ae50a9ea412809231239c7b75c027616c"}
Nov 02 14:00:24.394192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:24.394099    5034 scope.go:122] "RemoveContainer" containerID="3b2266365f0e6f1f2a38eca135ba6a389bd76d9d61cad39c68eb7a3dd50eb9d1"
Nov 02 14:00:25.185735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.185695    5034 generic.go:356] "Generic (PLEG): container finished" podID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerID="5d4b190f0ad5af39b702d7b48291072ae50a9ea412809231239c7b75c027616c" exitCode=60
Nov 02 14:00:25.186117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.186090    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" event={"ID":"ec5a92e1-a601-4634-93a5-3a7cbbb15168","Type":"ContainerDied","Data":"5d4b190f0ad5af39b702d7b48291072ae50a9ea412809231239c7b75c027616c"}
Nov 02 14:00:25.186302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.186287    5034 scope.go:122] "RemoveContainer" containerID="629f35db5d2d691f7b8caa99b157686ee112f4b81e12381dcc36763f6166b253"
Nov 02 14:00:25.187023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.186994    5034 scope.go:122] "RemoveContainer" containerID="5d4b190f0ad5af39b702d7b48291072ae50a9ea412809231239c7b75c027616c"
Nov 02 14:00:25.187585 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:25.187559    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw_odh-ai-gateway-infra(ec5a92e1-a601-4634-93a5-3a7cbbb15168)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168"
Nov 02 14:00:25.216157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.216124    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"]
Nov 02 14:00:25.267894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.257078    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh" event={"ID":"a886243d-85ee-4423-878e-4cf8a02772b3","Type":"ContainerStarted","Data":"7a010e5874e7c19edb3ef63f9a72cced6c90f662dda145b1602d3bdc8e4780be"}
Nov 02 14:00:25.308276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.308237    5034 generic.go:356] "Generic (PLEG): container finished" podID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerID="757e975cb2f79ce753c9674a8e9fa905636ff55d6ece222a03e45db4a0058606" exitCode=60
Nov 02 14:00:25.308603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.308557    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" event={"ID":"8c0b86e4-42b4-4c36-9691-78b40bffd94c","Type":"ContainerDied","Data":"757e975cb2f79ce753c9674a8e9fa905636ff55d6ece222a03e45db4a0058606"}
Nov 02 14:00:25.406806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.390462    5034 scope.go:122] "RemoveContainer" containerID="9ecc62c5941343fd6afa335777ffdc2bdbbf0dd0c1f7f24ea14e15778bb73451"
Nov 02 14:00:25.545823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.541936    5034 scope.go:122] "RemoveContainer" containerID="d51d3437a353448d0da3dac913b7594f847d3a77211f9646e5053ecbc82d8931"
Nov 02 14:00:25.641905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.635119    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"]
Nov 02 14:00:25.661805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.657762    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:25.738804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.737022    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:25.846337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:25.845937    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:26.049805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.029303    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"]
Nov 02 14:00:26.049805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.029448    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:26.078805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.062874    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:26.130865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.123860    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:26.204805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.203912    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:26.421134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.421090    5034 generic.go:356] "Generic (PLEG): container finished" podID="a886243d-85ee-4423-878e-4cf8a02772b3" containerID="7a010e5874e7c19edb3ef63f9a72cced6c90f662dda145b1602d3bdc8e4780be" exitCode=60
Nov 02 14:00:26.426895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.426851    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh" event={"ID":"a886243d-85ee-4423-878e-4cf8a02772b3","Type":"ContainerDied","Data":"7a010e5874e7c19edb3ef63f9a72cced6c90f662dda145b1602d3bdc8e4780be"}
Nov 02 14:00:26.427168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.427148    5034 scope.go:122] "RemoveContainer" containerID="3b2266365f0e6f1f2a38eca135ba6a389bd76d9d61cad39c68eb7a3dd50eb9d1"
Nov 02 14:00:26.484950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.484904    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:26.549806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.540078    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:26.702403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.659026    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:26.702403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.659473    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:00:26.778852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.736939    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:26.778852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.737345    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:26.778852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.755947    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:00:26.936842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.921326    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:26.936842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:26.922178    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:00:27.069803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.060463    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:27.173805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.124456    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:00:27.229567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.131827    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:27.238804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.234421    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:00:27.374361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.264689    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:00:27.374361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.264800    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:00:27.572274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.572225    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:27.607934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.581845    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:27.674899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.648089    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded"
Nov 02 14:00:27.674899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.651345    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:00:27.753105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.753063    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:27.837381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.753025    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:27.860500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.823534    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:27.999065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.947927    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:28.106107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:27.089771    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:28.106107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.061600    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:00:28.139566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.139524    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:28.139870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.139832    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:28.154264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.154218    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp" podUID="e71f9623-d37b-4294-a11b-a408ecc42eb0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:28.218439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.185510    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:00:28.366132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.365925    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:00:28.464600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.366439    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:00:28.464600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.306212    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:28.464600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.458924    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:00:28.464600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.459393    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:00:28.605843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.590067    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:00:28.605843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.595996    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:28.667922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.650392    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:00:28.682838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.681101    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0b86e4-42b4-4c36-9691-78b40bffd94c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0b86e4-42b4-4c36-9691-78b40bffd94c-maas-api-service-ca\") pod \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\" (UID: \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\") "
Nov 02 14:00:28.682838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.681359    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c0b86e4-42b4-4c36-9691-78b40bffd94c-kube-api-access-h58gz\" (UniqueName: \"kubernetes.io/projected/8c0b86e4-42b4-4c36-9691-78b40bffd94c-kube-api-access-h58gz\") pod \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\" (UID: \"8c0b86e4-42b4-4c36-9691-78b40bffd94c\") "
Nov 02 14:00:28.781883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.750374    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0b86e4-42b4-4c36-9691-78b40bffd94c-maas-api-service-ca" pod "8c0b86e4-42b4-4c36-9691-78b40bffd94c" (UID: "8c0b86e4-42b4-4c36-9691-78b40bffd94c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:28.939266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.868251    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:28.939266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.914060    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:28.940034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.939981    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:28.965150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:28.965087    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.082599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.082553    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.145805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.100524    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.145805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.122703    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c0b86e4-42b4-4c36-9691-78b40bffd94c-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:29.145805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.139359    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:29.145805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.145028    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.224924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.190592    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Nov 02 14:00:29.310804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.292296    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0b86e4-42b4-4c36-9691-78b40bffd94c-kube-api-access-h58gz" pod "8c0b86e4-42b4-4c36-9691-78b40bffd94c" (UID: "8c0b86e4-42b4-4c36-9691-78b40bffd94c"). InnerVolumeSpecName "kube-api-access-h58gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:29.331635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.331583    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-nw8tp" podUID="d345d401-090c-4d81-a5fc-fbf5ca938209" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.554187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.554150    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h58gz\" (UniqueName: \"kubernetes.io/projected/8c0b86e4-42b4-4c36-9691-78b40bffd94c-kube-api-access-h58gz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:29.554622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.554586    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.568422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.568374    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.687138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.687074    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:00:29.742866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.703090    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:00:29.914116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.914007    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9f9xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:29.914425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.914375    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:29.990016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:29.989951    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:30.025925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.025875    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:30.144221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.007769    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:00:30.144221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.034982    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:30.144221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.051668    5034 scope.go:122] "RemoveContainer" containerID="9d313a53151389d63f64cf97d6e5d25f6e857a7d07b3f08f471a8db2820feb1f"
Nov 02 14:00:30.179052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.178936    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:30.368434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.368389    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:30.369065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.369032    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:00:30.753896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.667853    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:30.789815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.780539    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:30.841605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.828043    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:00:30.949796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.949507    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": context deadline exceeded"
Nov 02 14:00:30.949796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.949629    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:30.987809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.984463    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5a92e1-a601-4634-93a5-3a7cbbb15168-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5a92e1-a601-4634-93a5-3a7cbbb15168-maas-api-service-ca\") pod \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\" (UID: \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\") "
Nov 02 14:00:30.987809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:30.984573    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5a92e1-a601-4634-93a5-3a7cbbb15168-kube-api-access-vqrf4\" (UniqueName: \"kubernetes.io/projected/ec5a92e1-a601-4634-93a5-3a7cbbb15168-kube-api-access-vqrf4\") pod \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\" (UID: \"ec5a92e1-a601-4634-93a5-3a7cbbb15168\") "
Nov 02 14:00:31.030584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.029071    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.039803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.031220    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5a92e1-a601-4634-93a5-3a7cbbb15168-maas-api-service-ca" pod "ec5a92e1-a601-4634-93a5-3a7cbbb15168" (UID: "ec5a92e1-a601-4634-93a5-3a7cbbb15168"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:31.099836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.096248    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:00:31.099836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.096965    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.099836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.098465    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:31.119835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.101550    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5a92e1-a601-4634-93a5-3a7cbbb15168-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:31.168769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.144949    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw" podUID="2358bb25-5f5b-42a4-9678-852be7940e34" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.75:50052\" within 1s: context deadline exceeded"
Nov 02 14:00:31.169560 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.159936    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.170419 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:00:31.160550    5034 logging.go:55] [core] [Channel #782 SubChannel #783]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.75:50052", ServerName: "10.132.0.75:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.75:50052: operation was canceled"
Nov 02 14:00:31.204834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.180383    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5a92e1-a601-4634-93a5-3a7cbbb15168-kube-api-access-vqrf4" pod "ec5a92e1-a601-4634-93a5-3a7cbbb15168" (UID: "ec5a92e1-a601-4634-93a5-3a7cbbb15168"). InnerVolumeSpecName "kube-api-access-vqrf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:31.204834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.192191    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:31.298571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.298473    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a886243d-85ee-4423-878e-4cf8a02772b3-kube-api-access-r779b\" (UniqueName: \"kubernetes.io/projected/a886243d-85ee-4423-878e-4cf8a02772b3-kube-api-access-r779b\") pod \"a886243d-85ee-4423-878e-4cf8a02772b3\" (UID: \"a886243d-85ee-4423-878e-4cf8a02772b3\") "
Nov 02 14:00:31.298997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.298971    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a886243d-85ee-4423-878e-4cf8a02772b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a886243d-85ee-4423-878e-4cf8a02772b3-maas-api-service-ca\") pod \"a886243d-85ee-4423-878e-4cf8a02772b3\" (UID: \"a886243d-85ee-4423-878e-4cf8a02772b3\") "
Nov 02 14:00:31.299889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.299863    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqrf4\" (UniqueName: \"kubernetes.io/projected/ec5a92e1-a601-4634-93a5-3a7cbbb15168-kube-api-access-vqrf4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:31.422833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.390962    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.424424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.401004    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.435810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.426041    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a886243d-85ee-4423-878e-4cf8a02772b3-maas-api-service-ca" pod "a886243d-85ee-4423-878e-4cf8a02772b3" (UID: "a886243d-85ee-4423-878e-4cf8a02772b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:31.446809 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:31.441542    5034 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.867s"
Nov 02 14:00:31.491846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.466062    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:00:31.555051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.454270    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.569971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.520873    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a886243d-85ee-4423-878e-4cf8a02772b3-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:31.570171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.454128    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.570309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.550682    5034 generic.go:356] "Generic (PLEG): container finished" podID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerID="aea5d9be618c8a253d09cef3b699196f1664325c9476bbfa1f6c2c61f265affc" exitCode=60
Nov 02 14:00:31.844898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.840799    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.847005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.846963    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.978802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.962090    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:31.998803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:31.990271    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"
Nov 02 14:00:32.206859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.201892    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:32.211016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.210936    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.219234    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.219624    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" event={"ID":"20200d2b-c009-4cee-86df-d8c05b8ca507","Type":"ContainerDied","Data":"aea5d9be618c8a253d09cef3b699196f1664325c9476bbfa1f6c2c61f265affc"}
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.219710    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"]
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220657    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220675    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220689    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e8643d5d-ca1c-4c95-9281-5f1e48b77ddd" containerName="revoke-keys"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220697    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8643d5d-ca1c-4c95-9281-5f1e48b77ddd" containerName="revoke-keys"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220716    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220723    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220735    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220741    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220753    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220760    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220808    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220817    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220864    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220872    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220903    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.220911    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.221090    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.221106    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.221114    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8643d5d-ca1c-4c95-9281-5f1e48b77ddd" containerName="revoke-keys"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.221128    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.221138    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.221156    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" containerName="cleanup"
Nov 02 14:00:32.224185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.221168    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerName="cleanup"
Nov 02 14:00:32.266864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.262004    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:32.266864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.262837    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:32.298017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.297962    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:32.322209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.322180    5034 scope.go:122] "RemoveContainer" containerID="fd9fda86d045da505547e3fdbba86e89657406ae1604914ba050e306254f99c7"
Nov 02 14:00:32.361108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.361081    5034 scope.go:122] "RemoveContainer" containerID="aea5d9be618c8a253d09cef3b699196f1664325c9476bbfa1f6c2c61f265affc"
Nov 02 14:00:32.385159 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:32.366579    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45_odh-ai-gateway-infra(20200d2b-c009-4cee-86df-d8c05b8ca507)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507"
Nov 02 14:00:32.405539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.375172    5034 scope.go:122] "RemoveContainer" containerID="9ecc62c5941343fd6afa335777ffdc2bdbbf0dd0c1f7f24ea14e15778bb73451"
Nov 02 14:00:32.406060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.406033    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"]
Nov 02 14:00:32.406332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.406314    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.465378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.427637    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a886243d-85ee-4423-878e-4cf8a02772b3-kube-api-access-r779b" pod "a886243d-85ee-4423-878e-4cf8a02772b3" (UID: "a886243d-85ee-4423-878e-4cf8a02772b3"). InnerVolumeSpecName "kube-api-access-r779b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:32.490123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.474312    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:32.509062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.492224    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:32.533879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.492661    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:32.554826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.535487    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4419c359-907e-4270-92a2-58980835b4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dea321a-fhjq4\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.633502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.633462    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4419c359-907e-4270-92a2-58980835b4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dea321a-fhjq4\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.633816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.633770    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zkb\" (UniqueName: \"kubernetes.io/projected/4419c359-907e-4270-92a2-58980835b4de-kube-api-access-z6zkb\") pod \"maas-api-revoke-sub-9dea321a-fhjq4\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.634300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.634284    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r779b\" (UniqueName: \"kubernetes.io/projected/a886243d-85ee-4423-878e-4cf8a02772b3-kube-api-access-r779b\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:32.706716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.679524    5034 scope.go:122] "RemoveContainer" containerID="6c8079f41f8b3eb8988f623f04d03530c77007ae0e5554d83c52475b23efaa80"
Nov 02 14:00:32.707316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.707278    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": context deadline exceeded"
Nov 02 14:00:32.757538 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.757507    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4419c359-907e-4270-92a2-58980835b4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dea321a-fhjq4\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.757845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.757819    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zkb\" (UniqueName: \"kubernetes.io/projected/4419c359-907e-4270-92a2-58980835b4de-kube-api-access-z6zkb\") pod \"maas-api-revoke-sub-9dea321a-fhjq4\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.785336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.785196    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4419c359-907e-4270-92a2-58980835b4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dea321a-fhjq4\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.791950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.791897    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4419c359-907e-4270-92a2-58980835b4de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dea321a-fhjq4\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.964108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.961994    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:32.964108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.962548    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:32.964108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.963591    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4419c359-907e-4270-92a2-58980835b4de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dea321a-fhjq4\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.964641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.964316    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zkb\" (UniqueName: \"kubernetes.io/projected/4419c359-907e-4270-92a2-58980835b4de-kube-api-access-z6zkb\") pod \"maas-api-revoke-sub-9dea321a-fhjq4\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:32.964951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.964910    5034 scope.go:122] "RemoveContainer" containerID="837c4ba3015ec426f06d965f1bac052f9de83e07884696de879081bc6b0643e7"
Nov 02 14:00:33.002862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:32.967871    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:33.145800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.130465    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:33.242971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.228982    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:00:33.306086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.305965    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:33.345841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.345252    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:33.348814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.347832    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:33.376981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.376925    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:00:33.531419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.527110    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:33.534393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.534338    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:33.671176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.671138    5034 scope.go:122] "RemoveContainer" containerID="bb132cead95c06ed4c52e460eda96a517fcd55788ffa1d8e4af34ae025adee44"
Nov 02 14:00:33.732802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.725409    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"
Nov 02 14:00:33.769495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.768921    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v" event={"ID":"8c0b86e4-42b4-4c36-9691-78b40bffd94c","Type":"ContainerDied","Data":"9837afb4f71923fa91edb367475b9c5730aa35f4226735048ce73ac8eb30878c"}
Nov 02 14:00:33.769495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:33.769040    5034 scope.go:122] "RemoveContainer" containerID="757e975cb2f79ce753c9674a8e9fa905636ff55d6ece222a03e45db4a0058606"
Nov 02 14:00:34.081802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.056801    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:00:34.192449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.082425    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:34.192710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.128272    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:34.192984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.130161    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:34.193082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.163905    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"
Nov 02 14:00:34.194390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.168840    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw" event={"ID":"ec5a92e1-a601-4634-93a5-3a7cbbb15168","Type":"ContainerDied","Data":"2153f2ab1e4c5cfa44fa18ebdc66b4462ba40832b89cb6188c4dd761f7a6f0bb"}
Nov 02 14:00:34.238023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.237968    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:34.248025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.243590    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"]
Nov 02 14:00:34.248025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.244555    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" containerName="cleanup"
Nov 02 14:00:34.248025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.244572    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" containerName="cleanup"
Nov 02 14:00:34.248025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.244624    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerName="cleanup"
Nov 02 14:00:34.248025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.244632    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerName="cleanup"
Nov 02 14:00:34.248025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.244825    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" containerName="cleanup"
Nov 02 14:00:34.248025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.244842    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" containerName="cleanup"
Nov 02 14:00:34.248025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.244852    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" containerName="cleanup"
Nov 02 14:00:34.278462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.277424    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:00:34.278462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.277463    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"]
Nov 02 14:00:34.278462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.277515    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:00:34.278462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.277546    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:00:34.283803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.283249    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:00:34.297181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.297147    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:00:34.302660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.301747    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:00:34.327002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.307192    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:34.327002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.324740    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.327002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.325264    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:00:34.328012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.327986    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:00:34.342257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.342182    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh" event={"ID":"a886243d-85ee-4423-878e-4cf8a02772b3","Type":"ContainerDied","Data":"e060394456dd151aa35ee71f4a0aa862d6dda1a157b9ebc4e4c5cbf94b9b6f2b"}
Nov 02 14:00:34.342469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.342456    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:00:34.356400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.356255    5034 scope.go:122] "RemoveContainer" containerID="5d4b190f0ad5af39b702d7b48291072ae50a9ea412809231239c7b75c027616c"
Nov 02 14:00:34.357006 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:34.356966    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5a92e1_a601_4634_93a5_3a7cbbb15168.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0b86e4_42b4_4c36_9691_78b40bffd94c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0b86e4_42b4_4c36_9691_78b40bffd94c.slice/crio-9837afb4f71923fa91edb367475b9c5730aa35f4226735048ce73ac8eb30878c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda886243d_85ee_4423_878e_4cf8a02772b3.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 14:00:34.359629 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.359606    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:00:34.367084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.367053    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"]
Nov 02 14:00:34.395922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.395894    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893800-hs8nh"]
Nov 02 14:00:34.398060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.397523    5034 scope.go:122] "RemoveContainer" containerID="7a010e5874e7c19edb3ef63f9a72cced6c90f662dda145b1602d3bdc8e4780be"
Nov 02 14:00:34.412039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.410899    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-463bdbfb-6w7fk\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.412039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.411032    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thxm5\" (UniqueName: \"kubernetes.io/projected/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-kube-api-access-thxm5\") pod \"maas-api-revoke-sub-463bdbfb-6w7fk\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.412039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.411079    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-463bdbfb-6w7fk\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.412039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.411845    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"]
Nov 02 14:00:34.437071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.436722    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"]
Nov 02 14:00:34.513483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.513442    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-463bdbfb-6w7fk\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.514104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.514065    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thxm5\" (UniqueName: \"kubernetes.io/projected/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-kube-api-access-thxm5\") pod \"maas-api-revoke-sub-463bdbfb-6w7fk\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.514343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.514309    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-463bdbfb-6w7fk\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.516381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.516356    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-463bdbfb-6w7fk\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.526318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.526281    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-463bdbfb-6w7fk\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.534384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.534347    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"]
Nov 02 14:00:34.543151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.543115    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893800-m9v9v"]
Nov 02 14:00:34.545551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.545525    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thxm5\" (UniqueName: \"kubernetes.io/projected/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-kube-api-access-thxm5\") pod \"maas-api-revoke-sub-463bdbfb-6w7fk\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.672320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.671980    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:34.680927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.678810    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"]
Nov 02 14:00:34.692530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.692505    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893800-52gfw"]
Nov 02 14:00:34.707838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.707807    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:34.818799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.818371    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20200d2b-c009-4cee-86df-d8c05b8ca507-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20200d2b-c009-4cee-86df-d8c05b8ca507-maas-api-service-ca\") pod \"20200d2b-c009-4cee-86df-d8c05b8ca507\" (UID: \"20200d2b-c009-4cee-86df-d8c05b8ca507\") "
Nov 02 14:00:34.818799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.818416    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20200d2b-c009-4cee-86df-d8c05b8ca507-kube-api-access-rmlpc\" (UniqueName: \"kubernetes.io/projected/20200d2b-c009-4cee-86df-d8c05b8ca507-kube-api-access-rmlpc\") pod \"20200d2b-c009-4cee-86df-d8c05b8ca507\" (UID: \"20200d2b-c009-4cee-86df-d8c05b8ca507\") "
Nov 02 14:00:34.820830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.819821    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20200d2b-c009-4cee-86df-d8c05b8ca507-maas-api-service-ca" pod "20200d2b-c009-4cee-86df-d8c05b8ca507" (UID: "20200d2b-c009-4cee-86df-d8c05b8ca507"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:34.822407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.822212    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20200d2b-c009-4cee-86df-d8c05b8ca507-kube-api-access-rmlpc" pod "20200d2b-c009-4cee-86df-d8c05b8ca507" (UID: "20200d2b-c009-4cee-86df-d8c05b8ca507"). InnerVolumeSpecName "kube-api-access-rmlpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:34.921372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.921325    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20200d2b-c009-4cee-86df-d8c05b8ca507-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:34.921372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:34.921352    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmlpc\" (UniqueName: \"kubernetes.io/projected/20200d2b-c009-4cee-86df-d8c05b8ca507-kube-api-access-rmlpc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:35.038330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.038294    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"]
Nov 02 14:00:35.369742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.369599    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk" event={"ID":"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70","Type":"ContainerStarted","Data":"289d460c23798122290648894ca51248f5444f6bb55e9bba2b4956b4b96e5efe"}
Nov 02 14:00:35.376289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.376261    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"
Nov 02 14:00:35.380529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.376527    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45" event={"ID":"20200d2b-c009-4cee-86df-d8c05b8ca507","Type":"ContainerDied","Data":"48e7226a14569053cb9f0efc0b2520749d68fbd1782df0168cea4a810294559f"}
Nov 02 14:00:35.380679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.380589    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4" event={"ID":"4419c359-907e-4270-92a2-58980835b4de","Type":"ContainerStarted","Data":"6cb96cf3bf5e9ca3997750ce3665e8b1114930e674cbc83f2aea8e49feb90f59"}
Nov 02 14:00:35.380679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.380628    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4" event={"ID":"4419c359-907e-4270-92a2-58980835b4de","Type":"ContainerStarted","Data":"d40766fa6f946e9c17c9fc7eb00bb90edaff3712fb47514018f487740256fa3f"}
Nov 02 14:00:35.380679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.380665    5034 scope.go:122] "RemoveContainer" containerID="aea5d9be618c8a253d09cef3b699196f1664325c9476bbfa1f6c2c61f265affc"
Nov 02 14:00:35.432956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.432917    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0b86e4-42b4-4c36-9691-78b40bffd94c" path="/var/lib/kubelet/pods/8c0b86e4-42b4-4c36-9691-78b40bffd94c/volumes"
Nov 02 14:00:35.433483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.433454    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a886243d-85ee-4423-878e-4cf8a02772b3" path="/var/lib/kubelet/pods/a886243d-85ee-4423-878e-4cf8a02772b3/volumes"
Nov 02 14:00:35.434011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.433993    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5a92e1-a601-4634-93a5-3a7cbbb15168" path="/var/lib/kubelet/pods/ec5a92e1-a601-4634-93a5-3a7cbbb15168/volumes"
Nov 02 14:00:35.434806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.434725    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4" podStartSLOduration=7.434708605 podStartE2EDuration="7.434708605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:35.43323553 +0000 UTC m=+1026.184539969" watchObservedRunningTime="2026-11-02 14:00:35.434708605 +0000 UTC m=+1026.186013043"
Nov 02 14:00:35.548646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.548614    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"]
Nov 02 14:00:35.552930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:35.552862    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893800-bgv45"]
Nov 02 14:00:36.390464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:36.390427    5034 generic.go:356] "Generic (PLEG): container finished" podID="4419c359-907e-4270-92a2-58980835b4de" containerID="6cb96cf3bf5e9ca3997750ce3665e8b1114930e674cbc83f2aea8e49feb90f59" exitCode=0
Nov 02 14:00:36.390970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:36.390518    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4" event={"ID":"4419c359-907e-4270-92a2-58980835b4de","Type":"ContainerDied","Data":"6cb96cf3bf5e9ca3997750ce3665e8b1114930e674cbc83f2aea8e49feb90f59"}
Nov 02 14:00:36.392469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:36.392441    5034 generic.go:356] "Generic (PLEG): container finished" podID="7d2bcce5-b059-48ec-8fd4-9eb78afa1d70" containerID="ae48816bebe372492ce321a89b81b51ee6e22bbfc8544cafcbf22cb00a2aaa63" exitCode=0
Nov 02 14:00:36.392597 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:36.392524    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk" event={"ID":"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70","Type":"ContainerDied","Data":"ae48816bebe372492ce321a89b81b51ee6e22bbfc8544cafcbf22cb00a2aaa63"}
Nov 02 14:00:37.001043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.001007    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"]
Nov 02 14:00:37.001657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.001642    5034 cpu_manager.go:414] "containerMap: removing container" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerName="cleanup"
Nov 02 14:00:37.001725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.001657    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerName="cleanup"
Nov 02 14:00:37.001725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.001677    5034 cpu_manager.go:414] "containerMap: removing container" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerName="cleanup"
Nov 02 14:00:37.001725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.001682    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerName="cleanup"
Nov 02 14:00:37.001893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.001824    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerName="cleanup"
Nov 02 14:00:37.001893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.001834    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerName="cleanup"
Nov 02 14:00:37.005989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.005960    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:37.028469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.027670    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d776122a-dockercfg-cjq2d\""
Nov 02 14:00:37.041344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.041308    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d776122a\""
Nov 02 14:00:37.051262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.051223    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"]
Nov 02 14:00:37.219703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.219308    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867a30da-620a-4c25-a569-c300147d5264-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf\" (UID: \"867a30da-620a-4c25-a569-c300147d5264\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:37.219703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.219370    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzvsq\" (UniqueName: \"kubernetes.io/projected/867a30da-620a-4c25-a569-c300147d5264-kube-api-access-dzvsq\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf\" (UID: \"867a30da-620a-4c25-a569-c300147d5264\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:37.320617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.320533    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867a30da-620a-4c25-a569-c300147d5264-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf\" (UID: \"867a30da-620a-4c25-a569-c300147d5264\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:37.320804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.320645    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzvsq\" (UniqueName: \"kubernetes.io/projected/867a30da-620a-4c25-a569-c300147d5264-kube-api-access-dzvsq\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf\" (UID: \"867a30da-620a-4c25-a569-c300147d5264\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:37.321381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.321361    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867a30da-620a-4c25-a569-c300147d5264-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf\" (UID: \"867a30da-620a-4c25-a569-c300147d5264\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:37.371461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.369701    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzvsq\" (UniqueName: \"kubernetes.io/projected/867a30da-620a-4c25-a569-c300147d5264-kube-api-access-dzvsq\") pod \"payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf\" (UID: \"867a30da-620a-4c25-a569-c300147d5264\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:37.398622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.398580    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" path="/var/lib/kubelet/pods/20200d2b-c009-4cee-86df-d8c05b8ca507/volumes"
Nov 02 14:00:37.627300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.627265    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:37.658659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.658282    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:37.783504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.772974    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:37.895168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.894517    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4419c359-907e-4270-92a2-58980835b4de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4419c359-907e-4270-92a2-58980835b4de-maas-api-xks-ca\") pod \"4419c359-907e-4270-92a2-58980835b4de\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") "
Nov 02 14:00:37.895168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.894588    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-kube-api-access-thxm5\" (UniqueName: \"kubernetes.io/projected/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-kube-api-access-thxm5\") pod \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") "
Nov 02 14:00:37.895168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.894763    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-xks-ca\") pod \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") "
Nov 02 14:00:37.895168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.894904    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4419c359-907e-4270-92a2-58980835b4de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4419c359-907e-4270-92a2-58980835b4de-maas-api-openshift-ca\") pod \"4419c359-907e-4270-92a2-58980835b4de\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") "
Nov 02 14:00:37.895975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.895644    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4419c359-907e-4270-92a2-58980835b4de-maas-api-openshift-ca" pod "4419c359-907e-4270-92a2-58980835b4de" (UID: "4419c359-907e-4270-92a2-58980835b4de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:37.905300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.904135    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-openshift-ca\") pod \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\" (UID: \"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70\") "
Nov 02 14:00:37.905300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.904226    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4419c359-907e-4270-92a2-58980835b4de-kube-api-access-z6zkb\" (UniqueName: \"kubernetes.io/projected/4419c359-907e-4270-92a2-58980835b4de-kube-api-access-z6zkb\") pod \"4419c359-907e-4270-92a2-58980835b4de\" (UID: \"4419c359-907e-4270-92a2-58980835b4de\") "
Nov 02 14:00:37.905300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.904969    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4419c359-907e-4270-92a2-58980835b4de-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:37.906010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.905899    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-openshift-ca" pod "7d2bcce5-b059-48ec-8fd4-9eb78afa1d70" (UID: "7d2bcce5-b059-48ec-8fd4-9eb78afa1d70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:37.926459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.923946    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-xks-ca" pod "7d2bcce5-b059-48ec-8fd4-9eb78afa1d70" (UID: "7d2bcce5-b059-48ec-8fd4-9eb78afa1d70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:37.926459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.923975    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4419c359-907e-4270-92a2-58980835b4de-kube-api-access-z6zkb" pod "4419c359-907e-4270-92a2-58980835b4de" (UID: "4419c359-907e-4270-92a2-58980835b4de"). InnerVolumeSpecName "kube-api-access-z6zkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:37.926459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.924039    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4419c359-907e-4270-92a2-58980835b4de-maas-api-xks-ca" pod "4419c359-907e-4270-92a2-58980835b4de" (UID: "4419c359-907e-4270-92a2-58980835b4de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:37.929113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:37.929074    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-kube-api-access-thxm5" pod "7d2bcce5-b059-48ec-8fd4-9eb78afa1d70" (UID: "7d2bcce5-b059-48ec-8fd4-9eb78afa1d70"). InnerVolumeSpecName "kube-api-access-thxm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:38.014225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.014183    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:38.014225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.014224    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:38.014465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.014240    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6zkb\" (UniqueName: \"kubernetes.io/projected/4419c359-907e-4270-92a2-58980835b4de-kube-api-access-z6zkb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:38.014465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.014254    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4419c359-907e-4270-92a2-58980835b4de-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:38.014465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.014266    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thxm5\" (UniqueName: \"kubernetes.io/projected/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70-kube-api-access-thxm5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:38.217828 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:00:38.217792    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867a30da_620a_4c25_a569_c300147d5264.slice/crio-8be3832243f250439b094c71d2be8e8e59a98aef2a96b925e185b88792f36fdf WatchSource:0}: Error finding container 8be3832243f250439b094c71d2be8e8e59a98aef2a96b925e185b88792f36fdf: Status 404 returned error can't find the container with id 8be3832243f250439b094c71d2be8e8e59a98aef2a96b925e185b88792f36fdf
Nov 02 14:00:38.219719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.219689    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"]
Nov 02 14:00:38.417860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.416362    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"
Nov 02 14:00:38.417860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.416412    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk" event={"ID":"7d2bcce5-b059-48ec-8fd4-9eb78afa1d70","Type":"ContainerDied","Data":"289d460c23798122290648894ca51248f5444f6bb55e9bba2b4956b4b96e5efe"}
Nov 02 14:00:38.417860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.416456    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="289d460c23798122290648894ca51248f5444f6bb55e9bba2b4956b4b96e5efe"
Nov 02 14:00:38.418393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.417983    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf" event={"ID":"867a30da-620a-4c25-a569-c300147d5264","Type":"ContainerStarted","Data":"b37a75f8f4dc80a9104ef782db157a255c0f750a11605d26d8c24ee00a6bc6de"}
Nov 02 14:00:38.418393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.418020    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf" event={"ID":"867a30da-620a-4c25-a569-c300147d5264","Type":"ContainerStarted","Data":"8be3832243f250439b094c71d2be8e8e59a98aef2a96b925e185b88792f36fdf"}
Nov 02 14:00:38.418798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.418767    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:38.420280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.420258    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4" event={"ID":"4419c359-907e-4270-92a2-58980835b4de","Type":"ContainerDied","Data":"d40766fa6f946e9c17c9fc7eb00bb90edaff3712fb47514018f487740256fa3f"}
Nov 02 14:00:38.420391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.420292    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d40766fa6f946e9c17c9fc7eb00bb90edaff3712fb47514018f487740256fa3f"
Nov 02 14:00:38.420391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.420334    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"
Nov 02 14:00:38.466870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.465249    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf" podStartSLOduration=2.465229548 podStartE2EDuration="2.465229548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:38.463374686 +0000 UTC m=+1029.214679123" watchObservedRunningTime="2026-11-02 14:00:38.465229548 +0000 UTC m=+1029.216533986"
Nov 02 14:00:38.990162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.990125    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"]
Nov 02 14:00:38.990857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.990835    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4419c359-907e-4270-92a2-58980835b4de" containerName="revoke-keys"
Nov 02 14:00:38.990857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.990854    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4419c359-907e-4270-92a2-58980835b4de" containerName="revoke-keys"
Nov 02 14:00:38.991015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.990877    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7d2bcce5-b059-48ec-8fd4-9eb78afa1d70" containerName="revoke-keys"
Nov 02 14:00:38.991015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.990882    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2bcce5-b059-48ec-8fd4-9eb78afa1d70" containerName="revoke-keys"
Nov 02 14:00:38.991015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.990904    5034 cpu_manager.go:414] "containerMap: removing container" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerName="cleanup"
Nov 02 14:00:38.991015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.990910    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerName="cleanup"
Nov 02 14:00:38.991144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.991060    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4419c359-907e-4270-92a2-58980835b4de" containerName="revoke-keys"
Nov 02 14:00:38.991144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.991070    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2bcce5-b059-48ec-8fd4-9eb78afa1d70" containerName="revoke-keys"
Nov 02 14:00:38.991144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.991080    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="20200d2b-c009-4cee-86df-d8c05b8ca507" containerName="cleanup"
Nov 02 14:00:38.994979 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:38.994961    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.021973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.021938    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"]
Nov 02 14:00:39.027737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.027702    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:39.027737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.027736    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:00:39.027963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.027735    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:00:39.034881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.034843    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d8234d-577w5\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.035088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.035057    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d8234d-577w5\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.035194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.035175    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b42dk\" (UniqueName: \"kubernetes.io/projected/97adbee5-6ae1-4e63-a452-ae747e102d3c-kube-api-access-b42dk\") pod \"maas-api-revoke-sub-79d8234d-577w5\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.037752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.037729    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:00:39.137112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.137080    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d8234d-577w5\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.137286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.137177    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d8234d-577w5\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.137286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.137237    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42dk\" (UniqueName: \"kubernetes.io/projected/97adbee5-6ae1-4e63-a452-ae747e102d3c-kube-api-access-b42dk\") pod \"maas-api-revoke-sub-79d8234d-577w5\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.137938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.137917    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79d8234d-577w5\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.139584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.139561    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79d8234d-577w5\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.163125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.163091    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42dk\" (UniqueName: \"kubernetes.io/projected/97adbee5-6ae1-4e63-a452-ae747e102d3c-kube-api-access-b42dk\") pod \"maas-api-revoke-sub-79d8234d-577w5\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.312085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.311979    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:39.628470 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:00:39.628435    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97adbee5_6ae1_4e63_a452_ae747e102d3c.slice/crio-d1ff21579d8101408fd17bd98ccb585dbc2888c0b1c08aec2aa3d0daf5cb5ba4 WatchSource:0}: Error finding container d1ff21579d8101408fd17bd98ccb585dbc2888c0b1c08aec2aa3d0daf5cb5ba4: Status 404 returned error can't find the container with id d1ff21579d8101408fd17bd98ccb585dbc2888c0b1c08aec2aa3d0daf5cb5ba4
Nov 02 14:00:39.631604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:39.631546    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"]
Nov 02 14:00:40.449239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:40.449138    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5" event={"ID":"97adbee5-6ae1-4e63-a452-ae747e102d3c","Type":"ContainerStarted","Data":"df1a858da76e1b3475b1e7f8f5b290f856fd68de96ee92545fed872e9f9c17c9"}
Nov 02 14:00:40.449239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:40.449208    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5" event={"ID":"97adbee5-6ae1-4e63-a452-ae747e102d3c","Type":"ContainerStarted","Data":"d1ff21579d8101408fd17bd98ccb585dbc2888c0b1c08aec2aa3d0daf5cb5ba4"}
Nov 02 14:00:40.494496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:40.494442    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5" podStartSLOduration=2.494427945 podStartE2EDuration="2.494427945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:40.492840723 +0000 UTC m=+1031.244145160" watchObservedRunningTime="2026-11-02 14:00:40.494427945 +0000 UTC m=+1031.245732480"
Nov 02 14:00:41.459686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:41.459651    5034 generic.go:356] "Generic (PLEG): container finished" podID="97adbee5-6ae1-4e63-a452-ae747e102d3c" containerID="df1a858da76e1b3475b1e7f8f5b290f856fd68de96ee92545fed872e9f9c17c9" exitCode=0
Nov 02 14:00:41.460111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:41.459742    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5" event={"ID":"97adbee5-6ae1-4e63-a452-ae747e102d3c","Type":"ContainerDied","Data":"df1a858da76e1b3475b1e7f8f5b290f856fd68de96ee92545fed872e9f9c17c9"}
Nov 02 14:00:42.686946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.686908    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:42.807118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.807082    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-xks-ca\") pod \"97adbee5-6ae1-4e63-a452-ae747e102d3c\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") "
Nov 02 14:00:42.807381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.807368    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-openshift-ca\") pod \"97adbee5-6ae1-4e63-a452-ae747e102d3c\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") "
Nov 02 14:00:42.807436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.807398    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97adbee5-6ae1-4e63-a452-ae747e102d3c-kube-api-access-b42dk\" (UniqueName: \"kubernetes.io/projected/97adbee5-6ae1-4e63-a452-ae747e102d3c-kube-api-access-b42dk\") pod \"97adbee5-6ae1-4e63-a452-ae747e102d3c\" (UID: \"97adbee5-6ae1-4e63-a452-ae747e102d3c\") "
Nov 02 14:00:42.807812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.807768    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-openshift-ca" pod "97adbee5-6ae1-4e63-a452-ae747e102d3c" (UID: "97adbee5-6ae1-4e63-a452-ae747e102d3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:42.809319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.809288    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-xks-ca" pod "97adbee5-6ae1-4e63-a452-ae747e102d3c" (UID: "97adbee5-6ae1-4e63-a452-ae747e102d3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:42.809437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.809396    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97adbee5-6ae1-4e63-a452-ae747e102d3c-kube-api-access-b42dk" pod "97adbee5-6ae1-4e63-a452-ae747e102d3c" (UID: "97adbee5-6ae1-4e63-a452-ae747e102d3c"). InnerVolumeSpecName "kube-api-access-b42dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:42.909106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.909069    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:42.909106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.909102    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b42dk\" (UniqueName: \"kubernetes.io/projected/97adbee5-6ae1-4e63-a452-ae747e102d3c-kube-api-access-b42dk\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:42.909106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:42.909113    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97adbee5-6ae1-4e63-a452-ae747e102d3c-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:43.477232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.477198    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"
Nov 02 14:00:43.477232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.477221    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5" event={"ID":"97adbee5-6ae1-4e63-a452-ae747e102d3c","Type":"ContainerDied","Data":"d1ff21579d8101408fd17bd98ccb585dbc2888c0b1c08aec2aa3d0daf5cb5ba4"}
Nov 02 14:00:43.477453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.477271    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1ff21579d8101408fd17bd98ccb585dbc2888c0b1c08aec2aa3d0daf5cb5ba4"
Nov 02 14:00:43.576970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.576935    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"]
Nov 02 14:00:43.577989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.577969    5034 cpu_manager.go:414] "containerMap: removing container" podUID="97adbee5-6ae1-4e63-a452-ae747e102d3c" containerName="revoke-keys"
Nov 02 14:00:43.578045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.577992    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97adbee5-6ae1-4e63-a452-ae747e102d3c" containerName="revoke-keys"
Nov 02 14:00:43.578154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.578143    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="97adbee5-6ae1-4e63-a452-ae747e102d3c" containerName="revoke-keys"
Nov 02 14:00:43.582976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.582955    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.607991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.607957    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:00:43.607991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.607957    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:43.608257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.608018    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:00:43.608257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.608214    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:00:43.626899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.626862    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"]
Nov 02 14:00:43.721576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.721486    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b3b1647-n6f7j\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.721576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.721560    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b3b1647-n6f7j\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.722051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.721659    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7cp\" (UniqueName: \"kubernetes.io/projected/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-kube-api-access-hv7cp\") pod \"maas-api-revoke-sub-6b3b1647-n6f7j\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.823451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.823413    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b3b1647-n6f7j\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.823636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.823488    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b3b1647-n6f7j\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.823636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.823515    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7cp\" (UniqueName: \"kubernetes.io/projected/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-kube-api-access-hv7cp\") pod \"maas-api-revoke-sub-6b3b1647-n6f7j\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.824280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.824256    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b3b1647-n6f7j\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.825959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.825936    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b3b1647-n6f7j\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.848849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.848818    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7cp\" (UniqueName: \"kubernetes.io/projected/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-kube-api-access-hv7cp\") pod \"maas-api-revoke-sub-6b3b1647-n6f7j\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:43.897456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:43.897415    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:44.088708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.088676    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"]
Nov 02 14:00:44.093728 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.093694    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.138207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.138169    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"]
Nov 02 14:00:44.226663 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:00:44.226637    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc343511c_58e3_4966_ad2c_bd3a2a9db0bd.slice/crio-0481621dc301cf21d1cd9b00753dd0164230a9a93dc654d2034de269c74d9125 WatchSource:0}: Error finding container 0481621dc301cf21d1cd9b00753dd0164230a9a93dc654d2034de269c74d9125: Status 404 returned error can't find the container with id 0481621dc301cf21d1cd9b00753dd0164230a9a93dc654d2034de269c74d9125
Nov 02 14:00:44.230606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.230575    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.233809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.231255    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plvxf\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-kube-api-access-plvxf\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.233809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.231348    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.233809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.231431    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.233809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.231501    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.233809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.231554    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.233809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.231642    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.233809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.231713    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.233809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.231829    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.233809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.232188    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"]
Nov 02 14:00:44.333254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333212    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.333445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333278    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.333445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333314    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.333445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333350    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.333445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333402    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.333671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333448    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.333671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333529    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.333774 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333682    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.333774 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333685    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.333774 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.333711    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plvxf\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-kube-api-access-plvxf\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.334050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.334026    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.334131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.334055    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.334322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.334292    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.334409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.334390    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.335794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.335759    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.335923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.335905    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.358885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.358853    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.359023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.358949    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plvxf\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-kube-api-access-plvxf\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.410287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.410248    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:44.490616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.490575    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j" event={"ID":"c343511c-58e3-4966-ad2c-bd3a2a9db0bd","Type":"ContainerStarted","Data":"0481621dc301cf21d1cd9b00753dd0164230a9a93dc654d2034de269c74d9125"}
Nov 02 14:00:44.847447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.846747    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:00:44.847447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.846859    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:00:44.847447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.846908    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:00:44.848037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:44.847473    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"]
Nov 02 14:00:45.502503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:45.502463    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" event={"ID":"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3","Type":"ContainerStarted","Data":"08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf"}
Nov 02 14:00:45.502693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:45.502529    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" event={"ID":"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3","Type":"ContainerStarted","Data":"37a1fd5a0e198843d1b13ba25be31deb1ca521ef5119ab70bc7c827a53401a8b"}
Nov 02 14:00:45.504366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:45.504332    5034 generic.go:356] "Generic (PLEG): container finished" podID="c343511c-58e3-4966-ad2c-bd3a2a9db0bd" containerID="d8536952f9c1d19f47bc9c16f7aa567f7d302d3d04088dd7507133dfa8c7795d" exitCode=0
Nov 02 14:00:45.504506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:45.504413    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j" event={"ID":"c343511c-58e3-4966-ad2c-bd3a2a9db0bd","Type":"ContainerDied","Data":"d8536952f9c1d19f47bc9c16f7aa567f7d302d3d04088dd7507133dfa8c7795d"}
Nov 02 14:00:45.552849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:45.552753    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podStartSLOduration=1.55273461 podStartE2EDuration="1.55273461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:45.55070857 +0000 UTC m=+1036.302013020" watchObservedRunningTime="2026-11-02 14:00:45.55273461 +0000 UTC m=+1036.304039049"
Nov 02 14:00:46.413507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:46.413464    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:46.414975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:46.414933    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:46.415386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:46.415361    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:46.854525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:46.854498    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:47.039969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.039933    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-kube-api-access-hv7cp\" (UniqueName: \"kubernetes.io/projected/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-kube-api-access-hv7cp\") pod \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") "
Nov 02 14:00:47.040182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.040141    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-openshift-ca\") pod \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") "
Nov 02 14:00:47.040254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.040203    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-xks-ca\") pod \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\" (UID: \"c343511c-58e3-4966-ad2c-bd3a2a9db0bd\") "
Nov 02 14:00:47.040514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.040490    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-openshift-ca" pod "c343511c-58e3-4966-ad2c-bd3a2a9db0bd" (UID: "c343511c-58e3-4966-ad2c-bd3a2a9db0bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:47.040923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.040903    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:47.042272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.042245    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-xks-ca" pod "c343511c-58e3-4966-ad2c-bd3a2a9db0bd" (UID: "c343511c-58e3-4966-ad2c-bd3a2a9db0bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:47.042372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.042287    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-kube-api-access-hv7cp" pod "c343511c-58e3-4966-ad2c-bd3a2a9db0bd" (UID: "c343511c-58e3-4966-ad2c-bd3a2a9db0bd"). InnerVolumeSpecName "kube-api-access-hv7cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:47.142947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.142863    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv7cp\" (UniqueName: \"kubernetes.io/projected/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-kube-api-access-hv7cp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:47.142947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.142895    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c343511c-58e3-4966-ad2c-bd3a2a9db0bd-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:00:47.410710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.410605    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:47.410710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.410672    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:47.525010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.524975    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"
Nov 02 14:00:47.525410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.524975    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j" event={"ID":"c343511c-58e3-4966-ad2c-bd3a2a9db0bd","Type":"ContainerDied","Data":"0481621dc301cf21d1cd9b00753dd0164230a9a93dc654d2034de269c74d9125"}
Nov 02 14:00:47.525410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:47.525109    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0481621dc301cf21d1cd9b00753dd0164230a9a93dc654d2034de269c74d9125"
Nov 02 14:00:48.411111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:48.411061    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:48.411321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:48.411147    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:48.430675 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:00:48.430629    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:00:49.420802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:49.418551    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:49.420802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:49.418629    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:50.411564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:50.411527    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:50.411892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:50.411860    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:50.512317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:50.510948    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:00:50.602606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:50.601972    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:50.716863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:50.701080    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:50.867808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:50.856829    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:00:50.934798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:50.933892    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:50.959805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:50.958255    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:50.975893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:50.974005    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:51.014803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.013881    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:51.091803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.091301    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:00:51.119812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.116286    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:51.196801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.196020    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:51.220804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.218896    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:51.441813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.441709    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:51.470838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.456231    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:51.470838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.460444    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:00:51.497572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.496639    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:00:51.749805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.745588    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:00:51.859395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.859351    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:51.898589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.898557    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:00:51.898923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.898906    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:00:51.966821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.934237    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:51.966821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:51.935880    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:00:52.053864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.044477    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:52.053864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.045237    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.053864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.045544    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:00:52.053864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.045602    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:00:52.113910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.113855    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:00:52.114268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.114238    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.215385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.215349    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:00:52.215727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.215709    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:00:52.281811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.244647    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.594984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.594935    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:00:52.644813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.287745    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.644813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.498751    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.644813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.498820    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:52.644813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.596479    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.644813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.596574    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:00:52.644813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.461077    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:00:52.644813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.597812    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:00:52.697627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.697592    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:52.697893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.697863    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:52.701804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.697911    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.753305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.753256    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.907003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.906862    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:52.974826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:52.968211    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:53.076839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.074037    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:00:53.082069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.082023    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:00:53.209864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.203251    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:53.209864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.203059    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:00:53.586956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.586910    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:53.619824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.609535    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.657981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.617602    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.688921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.638519    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:53.696701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.670546    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:53.696982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.696950    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:53.706859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.702934    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.732121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.717885    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:53.732519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.732389    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.781280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.781238    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:53.949868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.922867    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:00:53.978938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:53.978886    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:54.054474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.054425    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.101464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.076904    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:00:54.101464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.100390    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.208862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.207641    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.209226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.209186    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.413970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.413932    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:54.414242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.414216    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:54.595107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.595061    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.661941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.661881    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.663683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.662173    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.797807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.792981    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.988891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.983013    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:54.991526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:54.990253    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:00:55.093834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.080945    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:55.118815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.115298    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:55.198965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.198894    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6cbvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:55.199276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.199217    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:55.230916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.230862    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:55.269868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.243028    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:55.288713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.288673    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-2rmgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:55.336950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.336900    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:55.474805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.452236    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:55.475114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.475072    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:55.651236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.651123    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:55.680804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.663305    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:00:55.692841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.684952    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:55.977864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.870852    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:00:55.989802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:55.986143    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"]
Nov 02 14:00:56.466011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.081243    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:56.466011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.180939    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:00:56.466011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.344761    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-hdb7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:00:56.466011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.457384    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:56.466011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.456106    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:00:56.466011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.394644    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:56.502852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.481240    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:56.502852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.481335    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:56.675812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.655635    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:56.868758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.676125    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:56.899688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.708509    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:56.899688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.722769    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c343511c-58e3-4966-ad2c-bd3a2a9db0bd" containerName="revoke-keys"
Nov 02 14:00:56.899688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.891842    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c343511c-58e3-4966-ad2c-bd3a2a9db0bd" containerName="revoke-keys"
Nov 02 14:00:56.899688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.747700    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:56.904408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.765643    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:00:56.935029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.906803    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c343511c-58e3-4966-ad2c-bd3a2a9db0bd" containerName="revoke-keys"
Nov 02 14:00:56.946871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.906860    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:56.991405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.991372    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"]
Nov 02 14:00:56.993214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:56.993193    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.024165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.024130    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:57.053805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.031361    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:00:57.054070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.036134    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:00:57.130031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.129931    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:00:57.201747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.201711    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f08491-n4t8s\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.217080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.217020    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f08491-n4t8s\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.228044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.227996    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcz9k\" (UniqueName: \"kubernetes.io/projected/4a442fa8-13ed-46ba-a609-97b2d5da5f39-kube-api-access-qcz9k\") pod \"maas-api-revoke-sub-e0f08491-n4t8s\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.335451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.335394    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f08491-n4t8s\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.352761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.352712    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f08491-n4t8s\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.354053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.354030    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcz9k\" (UniqueName: \"kubernetes.io/projected/4a442fa8-13ed-46ba-a609-97b2d5da5f39-kube-api-access-qcz9k\") pod \"maas-api-revoke-sub-e0f08491-n4t8s\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.354450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.353959    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f08491-n4t8s\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.389868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.378736    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f08491-n4t8s\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.415906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.415871    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcz9k\" (UniqueName: \"kubernetes.io/projected/4a442fa8-13ed-46ba-a609-97b2d5da5f39-kube-api-access-qcz9k\") pod \"maas-api-revoke-sub-e0f08491-n4t8s\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.417283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.417256    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body=
Nov 02 14:00:57.417508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.417480    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused"
Nov 02 14:00:57.446312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.446275    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:00:57.458174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.458083    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:00:57.461906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.461861    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:00:57.462581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.462124    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:57.483837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.462151    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded"
Nov 02 14:00:57.484124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.462178    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:57.763743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.763701    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded"
Nov 02 14:00:57.860742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.860696    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:00:57.885391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.885343    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:57.894398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.894356    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:00:57.899911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.899867    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:57.934925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.934872    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:57.967943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:57.967893    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:00:58.187921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.187822    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:00:58.414857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.413145    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:00:58.699874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.699817    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:58.710583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.710518    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:58.716279 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.715950    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:00:58.902832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.861197    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:58.902832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.901978    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:58.902832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.902082    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:58.957068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.957022    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:59.012107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:58.982365    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:00:59.036384 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:00:59.020880    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a442fa8_13ed_46ba_a609_97b2d5da5f39.slice/crio-618447bc839dc541b65be2ab1016206b0129e3193e67208b40468e291a8f3995 WatchSource:0}: Error finding container 618447bc839dc541b65be2ab1016206b0129e3193e67208b40468e291a8f3995: Status 404 returned error can't find the container with id 618447bc839dc541b65be2ab1016206b0129e3193e67208b40468e291a8f3995
Nov 02 14:00:59.050356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.050117    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:00:59.063831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.063760    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"]
Nov 02 14:00:59.148142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.148103    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"]
Nov 02 14:00:59.210052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.207208    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:00:59.210052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.207267    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:00:59.210052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.208171    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.240803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.232132    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:00:59.240803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.232205    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:00:59.248797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.242240    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:00:59.248797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.243247    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"]
Nov 02 14:00:59.298427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.298399    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:59.304190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.302729    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:59.306282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.306261    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:00:59.325646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.318093    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.325646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.318228    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.325646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.318296    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53cf1eb-35ca-4451-816a-f352391a736b-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.325646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.318366    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53cf1eb-35ca-4451-816a-f352391a736b-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.325646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.318415    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.325646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.318549    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz4nd\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-kube-api-access-kz4nd\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.325646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.318720    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.325646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.318920    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.325646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.319066    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.329978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.328626    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:00:59.345922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.338282    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:00:59.420292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.420249    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.420542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.420341    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.420542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.420420    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.420542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.420473    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.420542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.420535    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.420774 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.420566    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53cf1eb-35ca-4451-816a-f352391a736b-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.420774 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.420631    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53cf1eb-35ca-4451-816a-f352391a736b-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.420774 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.420669    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.420774 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.420724    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz4nd\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-kube-api-access-kz4nd\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.421861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.421810    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.422108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.422087    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.423594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.423556    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.424495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.424474    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53cf1eb-35ca-4451-816a-f352391a736b-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.424999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.424977    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.431490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.431455    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.433231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.433210    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53cf1eb-35ca-4451-816a-f352391a736b-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.481811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.478528    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.481811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.478580    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz4nd\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-kube-api-access-kz4nd\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:00:59.673079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:00:59.672743    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:01:00.038964 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:01:00.035932    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53cf1eb_35ca_4451_816a_f352391a736b.slice/crio-e5d75581d67a0396e7b65787b3ee9bc526b67f6f1ddb443d6bf268f8191d5d91 WatchSource:0}: Error finding container e5d75581d67a0396e7b65787b3ee9bc526b67f6f1ddb443d6bf268f8191d5d91: Status 404 returned error can't find the container with id e5d75581d67a0396e7b65787b3ee9bc526b67f6f1ddb443d6bf268f8191d5d91
Nov 02 14:01:00.044213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:00.044031    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:01:00.044213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:00.044158    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:01:00.044385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:00.044217    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:01:00.054252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:00.054217    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"]
Nov 02 14:01:00.114218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:00.113805    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" event={"ID":"c53cf1eb-35ca-4451-816a-f352391a736b","Type":"ContainerStarted","Data":"e5d75581d67a0396e7b65787b3ee9bc526b67f6f1ddb443d6bf268f8191d5d91"}
Nov 02 14:01:00.118374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:00.118344    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s" event={"ID":"4a442fa8-13ed-46ba-a609-97b2d5da5f39","Type":"ContainerStarted","Data":"c97b8d5d80ee7da1495756d7832c85c526519d63c2ca9998f07d8b069bf7fea1"}
Nov 02 14:01:00.118578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:00.118564    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s" event={"ID":"4a442fa8-13ed-46ba-a609-97b2d5da5f39","Type":"ContainerStarted","Data":"618447bc839dc541b65be2ab1016206b0129e3193e67208b40468e291a8f3995"}
Nov 02 14:01:00.205803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:00.197997    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s" podStartSLOduration=5.197974515 podStartE2EDuration="5.197974515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:00.189452663 +0000 UTC m=+1050.940757119" watchObservedRunningTime="2026-11-02 14:01:00.197974515 +0000 UTC m=+1050.949278959"
Nov 02 14:01:01.127749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:01.127711    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" event={"ID":"c53cf1eb-35ca-4451-816a-f352391a736b","Type":"ContainerStarted","Data":"ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1"}
Nov 02 14:01:01.129672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:01.129627    5034 generic.go:356] "Generic (PLEG): container finished" podID="4a442fa8-13ed-46ba-a609-97b2d5da5f39" containerID="c97b8d5d80ee7da1495756d7832c85c526519d63c2ca9998f07d8b069bf7fea1" exitCode=0
Nov 02 14:01:01.129861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:01.129661    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s" event={"ID":"4a442fa8-13ed-46ba-a609-97b2d5da5f39","Type":"ContainerDied","Data":"c97b8d5d80ee7da1495756d7832c85c526519d63c2ca9998f07d8b069bf7fea1"}
Nov 02 14:01:01.208115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:01.205661    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" podStartSLOduration=2.20564105 podStartE2EDuration="2.20564105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:01.203343591 +0000 UTC m=+1051.954648028" watchObservedRunningTime="2026-11-02 14:01:01.20564105 +0000 UTC m=+1051.956945490"
Nov 02 14:01:01.690802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:01.689772    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:01:01.690802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:01.689979    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 14:01:01.690802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:01.690027    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 14:01:02.408434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.408369    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:01:02.507429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.507394    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-xks-ca\") pod \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") "
Nov 02 14:01:02.507637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.507503    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a442fa8-13ed-46ba-a609-97b2d5da5f39-kube-api-access-qcz9k\" (UniqueName: \"kubernetes.io/projected/4a442fa8-13ed-46ba-a609-97b2d5da5f39-kube-api-access-qcz9k\") pod \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") "
Nov 02 14:01:02.507637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.507599    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-openshift-ca\") pod \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\" (UID: \"4a442fa8-13ed-46ba-a609-97b2d5da5f39\") "
Nov 02 14:01:02.508586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.508500    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-openshift-ca" pod "4a442fa8-13ed-46ba-a609-97b2d5da5f39" (UID: "4a442fa8-13ed-46ba-a609-97b2d5da5f39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:02.508710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.508658    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:01:02.511169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.511129    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a442fa8-13ed-46ba-a609-97b2d5da5f39-kube-api-access-qcz9k" pod "4a442fa8-13ed-46ba-a609-97b2d5da5f39" (UID: "4a442fa8-13ed-46ba-a609-97b2d5da5f39"). InnerVolumeSpecName "kube-api-access-qcz9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:02.512442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.512423    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-xks-ca" pod "4a442fa8-13ed-46ba-a609-97b2d5da5f39" (UID: "4a442fa8-13ed-46ba-a609-97b2d5da5f39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:02.615490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.614365    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcz9k\" (UniqueName: \"kubernetes.io/projected/4a442fa8-13ed-46ba-a609-97b2d5da5f39-kube-api-access-qcz9k\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:01:02.615490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.614401    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a442fa8-13ed-46ba-a609-97b2d5da5f39-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:01:02.688852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.675165    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 14:01:02.688852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:02.675249    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 14:01:03.161522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:03.161490    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"
Nov 02 14:01:03.161522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:03.161511    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s" event={"ID":"4a442fa8-13ed-46ba-a609-97b2d5da5f39","Type":"ContainerDied","Data":"618447bc839dc541b65be2ab1016206b0129e3193e67208b40468e291a8f3995"}
Nov 02 14:01:03.161753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:03.161565    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="618447bc839dc541b65be2ab1016206b0129e3193e67208b40468e291a8f3995"
Nov 02 14:01:03.685804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:03.679017    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 14:01:03.685804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:03.679078    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 14:01:04.698805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:04.689235    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 14:01:04.698805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:04.689303    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 14:01:05.676801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:05.674884    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 14:01:05.676801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:05.674947    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 14:01:06.674019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:06.673982    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Nov 02 14:01:06.674421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:06.674048    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Nov 02 14:01:07.808858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:07.808817    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:01:07.810101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:07.810078    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:01:07.813230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:07.813208    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:01:09.992972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:09.992924    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"]
Nov 02 14:01:09.994641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:09.994608    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4a442fa8-13ed-46ba-a609-97b2d5da5f39" containerName="revoke-keys"
Nov 02 14:01:09.994641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:09.994640    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a442fa8-13ed-46ba-a609-97b2d5da5f39" containerName="revoke-keys"
Nov 02 14:01:09.995290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:09.995265    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a442fa8-13ed-46ba-a609-97b2d5da5f39" containerName="revoke-keys"
Nov 02 14:01:10.004484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.004455    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.011494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.011459    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"]
Nov 02 14:01:10.023505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.023390    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:01:10.041899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.037037    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:01:10.041899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.037067    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:01:10.041899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.037116    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:01:10.127412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.127358    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcd6g\" (UniqueName: \"kubernetes.io/projected/2cba9546-58bd-4185-9c00-1d0043b08f41-kube-api-access-lcd6g\") pod \"maas-api-revoke-sub-34b1998f-clhxm\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.127619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.127594    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34b1998f-clhxm\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.127726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.127709    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34b1998f-clhxm\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.229210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.229171    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcd6g\" (UniqueName: \"kubernetes.io/projected/2cba9546-58bd-4185-9c00-1d0043b08f41-kube-api-access-lcd6g\") pod \"maas-api-revoke-sub-34b1998f-clhxm\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.229401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.229291    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34b1998f-clhxm\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.229401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.229375    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34b1998f-clhxm\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.230168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.230137    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34b1998f-clhxm\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.231849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.231822    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34b1998f-clhxm\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.254123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.254056    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcd6g\" (UniqueName: \"kubernetes.io/projected/2cba9546-58bd-4185-9c00-1d0043b08f41-kube-api-access-lcd6g\") pod \"maas-api-revoke-sub-34b1998f-clhxm\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.334872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.334839    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:10.679353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:10.679327    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"]
Nov 02 14:01:11.263527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:11.263490    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm" event={"ID":"2cba9546-58bd-4185-9c00-1d0043b08f41","Type":"ContainerStarted","Data":"1f05aa7b629a4d2edc37bfd9b231a3eab36320eac12bc57e84c26d16a02dacee"}
Nov 02 14:01:11.263969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:11.263556    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm" event={"ID":"2cba9546-58bd-4185-9c00-1d0043b08f41","Type":"ContainerStarted","Data":"ba074cd79aeed9e62fd7aa6f02c782c7cd60ffc5b1a73ca7cc606fd43995ef44"}
Nov 02 14:01:11.317198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:11.317122    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm" podStartSLOduration=2.317103188 podStartE2EDuration="2.317103188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:11.307645515 +0000 UTC m=+1062.058949954" watchObservedRunningTime="2026-11-02 14:01:11.317103188 +0000 UTC m=+1062.068407628"
Nov 02 14:01:12.275197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:12.275156    5034 generic.go:356] "Generic (PLEG): container finished" podID="2cba9546-58bd-4185-9c00-1d0043b08f41" containerID="1f05aa7b629a4d2edc37bfd9b231a3eab36320eac12bc57e84c26d16a02dacee" exitCode=0
Nov 02 14:01:12.275710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:12.275232    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm" event={"ID":"2cba9546-58bd-4185-9c00-1d0043b08f41","Type":"ContainerDied","Data":"1f05aa7b629a4d2edc37bfd9b231a3eab36320eac12bc57e84c26d16a02dacee"}
Nov 02 14:01:13.801969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:13.801913    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:13.943665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:13.943615    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Nov 02 14:01:14.086471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.086360    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:01:14.284868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.086706    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:01:14.285460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.145423    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:01:14.292866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.146677    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:01:14.293399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.227868    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:14.387083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.386961    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:14.596394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.596349    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:14.697926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.697876    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:14.716509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.716075    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:14.802929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.802879    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:14.804874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.804848    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:01:14.840810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.835553    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-xks-ca\") pod \"2cba9546-58bd-4185-9c00-1d0043b08f41\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") "
Nov 02 14:01:14.841131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.841100    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cba9546-58bd-4185-9c00-1d0043b08f41-kube-api-access-lcd6g\" (UniqueName: \"kubernetes.io/projected/2cba9546-58bd-4185-9c00-1d0043b08f41-kube-api-access-lcd6g\") pod \"2cba9546-58bd-4185-9c00-1d0043b08f41\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") "
Nov 02 14:01:14.841439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.841419    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-openshift-ca\") pod \"2cba9546-58bd-4185-9c00-1d0043b08f41\" (UID: \"2cba9546-58bd-4185-9c00-1d0043b08f41\") "
Nov 02 14:01:14.867126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.867086    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-openshift-ca" pod "2cba9546-58bd-4185-9c00-1d0043b08f41" (UID: "2cba9546-58bd-4185-9c00-1d0043b08f41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:14.931618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.927427    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cba9546-58bd-4185-9c00-1d0043b08f41-kube-api-access-lcd6g" pod "2cba9546-58bd-4185-9c00-1d0043b08f41" (UID: "2cba9546-58bd-4185-9c00-1d0043b08f41"). InnerVolumeSpecName "kube-api-access-lcd6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:14.933049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.933008    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-xks-ca" pod "2cba9546-58bd-4185-9c00-1d0043b08f41" (UID: "2cba9546-58bd-4185-9c00-1d0043b08f41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:14.973878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.948213    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:01:14.973878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.948255    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cba9546-58bd-4185-9c00-1d0043b08f41-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:01:14.973878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:14.948270    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcd6g\" (UniqueName: \"kubernetes.io/projected/2cba9546-58bd-4185-9c00-1d0043b08f41-kube-api-access-lcd6g\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:01:15.236915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.227275    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:15.290932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.290868    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:01:15.308106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.308071    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:01:15.308465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.308446    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"]
Nov 02 14:01:15.333805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.313212    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:15.344925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.344882    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:01:15.345254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.333914    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:01:15.345483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.333947    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:15.345919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.345896    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:01:15.346112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.346096    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:01:15.368510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.333874    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:15.390833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.390791    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:15.727804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.646580    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:01:15.793520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.700968    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:01:15.793520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.790030    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-639b514c-pkkzc"]
Nov 02 14:01:15.793520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.791978    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:01:15.793520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.792026    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:01:15.793520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.792063    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:01:15.793520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.792097    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:01:15.808107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.806880    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:15.986083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.986050    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm" event={"ID":"2cba9546-58bd-4185-9c00-1d0043b08f41","Type":"ContainerDied","Data":"ba074cd79aeed9e62fd7aa6f02c782c7cd60ffc5b1a73ca7cc606fd43995ef44"}
Nov 02 14:01:15.986627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.986606    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba074cd79aeed9e62fd7aa6f02c782c7cd60ffc5b1a73ca7cc606fd43995ef44"
Nov 02 14:01:15.986793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:15.986547    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"
Nov 02 14:01:16.170434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.170260    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:01:16.170434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.170333    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:16.170798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.170758    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:01:16.170929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.170907    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:16.264803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.239971    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:01:16.265129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.240986    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:01:16.267196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.267171    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:01:16.307814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.292865    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:16.352836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.351708    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:16.379813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.375712    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:01:16.379813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.375891    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:16.442917 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.442870    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:16.791045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.790988    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:16.885851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.876696    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:01:16.885851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:16.878717    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:01:17.264713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.264667    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:17.285092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.265315    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:17.285356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.265357    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:01:17.328957 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.328909    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:01:17.432807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.416582    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:01:17.463146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.463092    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:17.478498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.478450    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:01:17.490808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.489040    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:01:17.683408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.683309    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded"
Nov 02 14:01:17.843890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.843830    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:17.881018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.880957    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:17.892816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:17.883036    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:18.164800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.153655    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd07df3b-b54c-41d8-9c37-ad78d012f213" path="/var/lib/kubelet/pods/dd07df3b-b54c-41d8-9c37-ad78d012f213/volumes"
Nov 02 14:01:18.294761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.294708    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:18.340974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.340921    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:18.341867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.341841    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:01:18.349891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.342718    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:18.463993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.463885    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:01:18.465686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.464217    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:01:18.493536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.493493    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:18.653802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.647099    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:18.671802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.662139    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:01:18.765799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.762559    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:01:18.765799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.763661    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:01:18.868330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.868301    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:01:18.869679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.868718    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:18.914826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.894973    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:01:18.915771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:18.894984    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:19.187914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.182088    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:01:19.225571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.215933    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:19.416912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.416867    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:19.475299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.475219    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:01:19.475556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.475544    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:01:19.511516 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:01:19.511286    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:01:19.542647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.542431    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:01:19.543382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.543363    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:01:19.543579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.543568    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:01:19.635126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.635088    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"]
Nov 02 14:01:19.637090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.637057    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2cba9546-58bd-4185-9c00-1d0043b08f41" containerName="revoke-keys"
Nov 02 14:01:19.637090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.637088    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cba9546-58bd-4185-9c00-1d0043b08f41" containerName="revoke-keys"
Nov 02 14:01:19.637638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.637613    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cba9546-58bd-4185-9c00-1d0043b08f41" containerName="revoke-keys"
Nov 02 14:01:19.643825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.643766    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.662249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.662221    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4c6741-openshift-default-dockercfg-7gdkl\""
Nov 02 14:01:19.663320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.663297    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"]
Nov 02 14:01:19.745399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.745350    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-data\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.745598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.745434    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-kube-api-access-4wbhw\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.745598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.745573    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-podinfo\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.745725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.745683    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-certs\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.745826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.745765    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-credential-socket\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.745895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.745858    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istiod-ca-cert\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.745971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.745953    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-envoy\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.746023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.745996    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-socket\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.746129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.746114    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-token\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.848973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.848934    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-token\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.849163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849057    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-data\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.849163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849108    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-kube-api-access-4wbhw\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.849625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849183    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-podinfo\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.849625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849257    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-certs\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.849625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849314    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-credential-socket\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.849625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849369    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istiod-ca-cert\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.849625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849461    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-envoy\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.849625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849492    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-socket\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.849991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849958    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-data\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.850047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.849983    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-socket\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.850257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.850216    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-credential-socket\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.851219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.850678    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istiod-ca-cert\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.851219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.851169    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-certs\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.853273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.853249    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-podinfo\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.853495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.853462    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-envoy\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.884796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.881832    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-kube-api-access-4wbhw\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.898724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.896259    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-token\") pod \"e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") " pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:19.988220 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:19.988180    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:20.517456 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:01:20.517427    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34e96f8_c765_4ad6_8cd4_aec1f564ca78.slice/crio-ad2d108ebad4169b99cd0f79617bcfb72235b34a5e3fcf6647d668e4315c50e5 WatchSource:0}: Error finding container ad2d108ebad4169b99cd0f79617bcfb72235b34a5e3fcf6647d668e4315c50e5: Status 404 returned error can't find the container with id ad2d108ebad4169b99cd0f79617bcfb72235b34a5e3fcf6647d668e4315c50e5
Nov 02 14:01:20.519974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:20.519935    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:01:20.520046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:20.520003    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:01:20.520046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:20.520031    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:01:20.520828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:20.520800    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"]
Nov 02 14:01:20.554430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:20.554397    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" event={"ID":"f34e96f8-c765-4ad6-8cd4-aec1f564ca78","Type":"ContainerStarted","Data":"ad2d108ebad4169b99cd0f79617bcfb72235b34a5e3fcf6647d668e4315c50e5"}
Nov 02 14:01:21.566216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:21.566174    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" event={"ID":"f34e96f8-c765-4ad6-8cd4-aec1f564ca78","Type":"ContainerStarted","Data":"ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9"}
Nov 02 14:01:21.639690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:21.639634    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" podStartSLOduration=2.639616938 podStartE2EDuration="2.639616938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:21.636043312 +0000 UTC m=+1072.387347775" watchObservedRunningTime="2026-11-02 14:01:21.639616938 +0000 UTC m=+1072.390921375"
Nov 02 14:01:21.988835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:21.988792    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:21.995422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:21.995393    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:22.706811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:22.703971    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:22.714302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:22.714273    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:01:27.955867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:27.955827    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"]
Nov 02 14:01:27.961093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:27.961068    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:27.986105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:27.986078    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"]
Nov 02 14:01:27.987921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:27.987898    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9eb67b\""
Nov 02 14:01:27.988132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:27.988118    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9eb67b-dockercfg-lskzn\""
Nov 02 14:01:28.034762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.034721    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfef9a88-5782-442d-a126-25b39d968683-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx\" (UID: \"bfef9a88-5782-442d-a126-25b39d968683\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:28.035337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.035308    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbnsm\" (UniqueName: \"kubernetes.io/projected/bfef9a88-5782-442d-a126-25b39d968683-kube-api-access-sbnsm\") pod \"payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx\" (UID: \"bfef9a88-5782-442d-a126-25b39d968683\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:28.057451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.057426    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"]
Nov 02 14:01:28.062170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.062152    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:28.214855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.139891    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll\" (UID: \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\") " pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:28.215370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.215330    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbnsm\" (UniqueName: \"kubernetes.io/projected/bfef9a88-5782-442d-a126-25b39d968683-kube-api-access-sbnsm\") pod \"payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx\" (UID: \"bfef9a88-5782-442d-a126-25b39d968683\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:28.254880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.230508    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"]
Nov 02 14:01:28.255205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.250153    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbnsm\" (UniqueName: \"kubernetes.io/projected/bfef9a88-5782-442d-a126-25b39d968683-kube-api-access-sbnsm\") pod \"payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx\" (UID: \"bfef9a88-5782-442d-a126-25b39d968683\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:28.255361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.255089    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfef9a88-5782-442d-a126-25b39d968683-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx\" (UID: \"bfef9a88-5782-442d-a126-25b39d968683\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:28.255682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.255654    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjc5g\" (UniqueName: \"kubernetes.io/projected/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-kube-api-access-cjc5g\") pod \"payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll\" (UID: \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\") " pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:28.261801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.261702    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfef9a88-5782-442d-a126-25b39d968683-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx\" (UID: \"bfef9a88-5782-442d-a126-25b39d968683\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:28.298634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.298286    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:28.362843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.362808    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjc5g\" (UniqueName: \"kubernetes.io/projected/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-kube-api-access-cjc5g\") pod \"payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll\" (UID: \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\") " pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:28.363059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.362985    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll\" (UID: \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\") " pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:28.363754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.363730    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll\" (UID: \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\") " pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:28.393109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.388105    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjc5g\" (UniqueName: \"kubernetes.io/projected/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-kube-api-access-cjc5g\") pod \"payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll\" (UID: \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\") " pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:28.666705 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:01:28.666675    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfef9a88_5782_442d_a126_25b39d968683.slice/crio-1b0c331bb2bdd2c540c42bc48874eb8847efce5a3a94c4c157c1d09a1eb25163 WatchSource:0}: Error finding container 1b0c331bb2bdd2c540c42bc48874eb8847efce5a3a94c4c157c1d09a1eb25163: Status 404 returned error can't find the container with id 1b0c331bb2bdd2c540c42bc48874eb8847efce5a3a94c4c157c1d09a1eb25163
Nov 02 14:01:28.668896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.668867    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"]
Nov 02 14:01:28.680536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.680516    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:28.763143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.761769    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx" event={"ID":"bfef9a88-5782-442d-a126-25b39d968683","Type":"ContainerStarted","Data":"fdc55804b04714347e1c3ab32ce755eac34e31598a0b53fa34c2cfe60bfa0349"}
Nov 02 14:01:28.763143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.761855    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx" event={"ID":"bfef9a88-5782-442d-a126-25b39d968683","Type":"ContainerStarted","Data":"1b0c331bb2bdd2c540c42bc48874eb8847efce5a3a94c4c157c1d09a1eb25163"}
Nov 02 14:01:28.763143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.762711    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:28.817803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.817376    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx" podStartSLOduration=1.817354671 podStartE2EDuration="1.817354671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:28.810381019 +0000 UTC m=+1079.561685458" watchObservedRunningTime="2026-11-02 14:01:28.817354671 +0000 UTC m=+1079.568659110"
Nov 02 14:01:28.995702 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:01:28.995666    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc5e889_1ca4_4b79_a657_0cd1005f6b70.slice/crio-80116ee0aedf5103f1b19817f5274d4752eb88d95e153605b7546cba1e020cab WatchSource:0}: Error finding container 80116ee0aedf5103f1b19817f5274d4752eb88d95e153605b7546cba1e020cab: Status 404 returned error can't find the container with id 80116ee0aedf5103f1b19817f5274d4752eb88d95e153605b7546cba1e020cab
Nov 02 14:01:28.998967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:28.998939    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"]
Nov 02 14:01:29.771549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:29.771514    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll" event={"ID":"8dc5e889-1ca4-4b79-a657-0cd1005f6b70","Type":"ContainerStarted","Data":"f4319a84ac42d09d712264ed633499865d9f329de7874373aa4d4ec21ca3eaa2"}
Nov 02 14:01:29.771910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:29.771573    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll" event={"ID":"8dc5e889-1ca4-4b79-a657-0cd1005f6b70","Type":"ContainerStarted","Data":"80116ee0aedf5103f1b19817f5274d4752eb88d95e153605b7546cba1e020cab"}
Nov 02 14:01:29.771910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:29.771811    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:29.820811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:29.820725    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll" podStartSLOduration=1.8207106579999999 podStartE2EDuration="1.820710658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:29.815218111 +0000 UTC m=+1080.566522562" watchObservedRunningTime="2026-11-02 14:01:29.820710658 +0000 UTC m=+1080.572015096"
Nov 02 14:01:30.033490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:30.033405    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"]
Nov 02 14:01:30.038725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:30.038693    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0220407e-ltf74"]
Nov 02 14:01:31.398587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:31.398550    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77e9d2f-ac79-421c-a99d-ef56c7749d83" path="/var/lib/kubelet/pods/d77e9d2f-ac79-421c-a99d-ef56c7749d83/volumes"
Nov 02 14:01:38.034091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:38.034050    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"]
Nov 02 14:01:38.047690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:38.047653    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5e942a-z2jvk"]
Nov 02 14:01:38.052277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:38.052251    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"]
Nov 02 14:01:38.056522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:38.056496    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e10f9489-clqtr"]
Nov 02 14:01:39.030117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:39.030079    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"]
Nov 02 14:01:39.042928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:39.042902    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bbf4923-6t6ln"]
Nov 02 14:01:39.401021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:39.400925    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f7df55-c680-4385-b43e-0c5a0028c664" path="/var/lib/kubelet/pods/24f7df55-c680-4385-b43e-0c5a0028c664/volumes"
Nov 02 14:01:39.401852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:39.401817    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a020e3b-3fd5-4edb-bd21-a2dbe5f99773" path="/var/lib/kubelet/pods/4a020e3b-3fd5-4edb-bd21-a2dbe5f99773/volumes"
Nov 02 14:01:39.402763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:39.402737    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc14d42-c40b-45ee-950f-9fe330b69256" path="/var/lib/kubelet/pods/9dc14d42-c40b-45ee-950f-9fe330b69256/volumes"
Nov 02 14:01:40.790350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:40.790324    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:01:40.790753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:40.790388    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:01:48.415297 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:01:48.415258    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:01:53.034280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:53.034243    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"]
Nov 02 14:01:53.046577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:53.046546    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-523509a0-g62pk"]
Nov 02 14:01:53.413184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:53.413099    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db1517b-dbe2-4d18-b155-a87740ea7f75" path="/var/lib/kubelet/pods/3db1517b-dbe2-4d18-b155-a87740ea7f75/volumes"
Nov 02 14:01:54.806398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.806329    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"]
Nov 02 14:01:54.815887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.814589    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:54.815887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.814489    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"]
Nov 02 14:01:54.842632 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.842611    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:01:54.855762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.855738    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:01:54.856002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.855875    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:01:54.856139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.855931    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:01:54.926570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.926531    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnfp\" (UniqueName: \"kubernetes.io/projected/f7f1a70e-b375-49cc-b456-cf86a19733b2-kube-api-access-5qnfp\") pod \"maas-api-revoke-sub-e2cd0adc-nsftw\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:54.926774 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.926712    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2cd0adc-nsftw\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:54.927136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:54.927115    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2cd0adc-nsftw\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:55.028472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:55.028431    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnfp\" (UniqueName: \"kubernetes.io/projected/f7f1a70e-b375-49cc-b456-cf86a19733b2-kube-api-access-5qnfp\") pod \"maas-api-revoke-sub-e2cd0adc-nsftw\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:55.028656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:55.028569    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2cd0adc-nsftw\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:55.028716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:55.028698    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2cd0adc-nsftw\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:55.029439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:55.029416    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2cd0adc-nsftw\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:55.031103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:55.031071    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2cd0adc-nsftw\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:55.055119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:55.055094    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnfp\" (UniqueName: \"kubernetes.io/projected/f7f1a70e-b375-49cc-b456-cf86a19733b2-kube-api-access-5qnfp\") pod \"maas-api-revoke-sub-e2cd0adc-nsftw\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:55.137200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:55.137122    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:55.528545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:55.528515    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"]
Nov 02 14:01:56.123882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:56.123757    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw" event={"ID":"f7f1a70e-b375-49cc-b456-cf86a19733b2","Type":"ContainerStarted","Data":"769e230ed6bc58cc1df6f80c60339d44645fc48c1d86b17f64d4499e950f6bd5"}
Nov 02 14:01:56.123882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:56.123839    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw" event={"ID":"f7f1a70e-b375-49cc-b456-cf86a19733b2","Type":"ContainerStarted","Data":"8f06afc79eadda50216f56166e80a1ad263504923fe6c5b086b9a26d31928bb3"}
Nov 02 14:01:56.167560 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:56.167503    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw" podStartSLOduration=2.167487532 podStartE2EDuration="2.167487532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:56.166367301 +0000 UTC m=+1106.917671738" watchObservedRunningTime="2026-11-02 14:01:56.167487532 +0000 UTC m=+1106.918791969"
Nov 02 14:01:57.196635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:57.196537    5034 generic.go:356] "Generic (PLEG): container finished" podID="f7f1a70e-b375-49cc-b456-cf86a19733b2" containerID="769e230ed6bc58cc1df6f80c60339d44645fc48c1d86b17f64d4499e950f6bd5" exitCode=0
Nov 02 14:01:57.197446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:57.197415    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw" event={"ID":"f7f1a70e-b375-49cc-b456-cf86a19733b2","Type":"ContainerDied","Data":"769e230ed6bc58cc1df6f80c60339d44645fc48c1d86b17f64d4499e950f6bd5"}
Nov 02 14:01:58.445637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.445610    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:58.502386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.502355    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-openshift-ca\") pod \"f7f1a70e-b375-49cc-b456-cf86a19733b2\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") "
Nov 02 14:01:58.502826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.502765    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f1a70e-b375-49cc-b456-cf86a19733b2-kube-api-access-5qnfp\" (UniqueName: \"kubernetes.io/projected/f7f1a70e-b375-49cc-b456-cf86a19733b2-kube-api-access-5qnfp\") pod \"f7f1a70e-b375-49cc-b456-cf86a19733b2\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") "
Nov 02 14:01:58.503083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.503063    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-xks-ca\") pod \"f7f1a70e-b375-49cc-b456-cf86a19733b2\" (UID: \"f7f1a70e-b375-49cc-b456-cf86a19733b2\") "
Nov 02 14:01:58.504624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.502882    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-openshift-ca" pod "f7f1a70e-b375-49cc-b456-cf86a19733b2" (UID: "f7f1a70e-b375-49cc-b456-cf86a19733b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:58.530362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.530316    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-xks-ca" pod "f7f1a70e-b375-49cc-b456-cf86a19733b2" (UID: "f7f1a70e-b375-49cc-b456-cf86a19733b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:58.530362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.530349    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f1a70e-b375-49cc-b456-cf86a19733b2-kube-api-access-5qnfp" pod "f7f1a70e-b375-49cc-b456-cf86a19733b2" (UID: "f7f1a70e-b375-49cc-b456-cf86a19733b2"). InnerVolumeSpecName "kube-api-access-5qnfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:58.609966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.609913    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnfp\" (UniqueName: \"kubernetes.io/projected/f7f1a70e-b375-49cc-b456-cf86a19733b2-kube-api-access-5qnfp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:01:58.610254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.610223    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:01:58.610254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:58.610254    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1a70e-b375-49cc-b456-cf86a19733b2-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:01:59.223824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:59.223763    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw" event={"ID":"f7f1a70e-b375-49cc-b456-cf86a19733b2","Type":"ContainerDied","Data":"8f06afc79eadda50216f56166e80a1ad263504923fe6c5b086b9a26d31928bb3"}
Nov 02 14:01:59.223824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:59.223773    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"
Nov 02 14:01:59.224120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:01:59.223852    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f06afc79eadda50216f56166e80a1ad263504923fe6c5b086b9a26d31928bb3"
Nov 02 14:02:00.533128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.533095    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"]
Nov 02 14:02:00.534095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.534069    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f7f1a70e-b375-49cc-b456-cf86a19733b2" containerName="revoke-keys"
Nov 02 14:02:00.534095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.534092    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f1a70e-b375-49cc-b456-cf86a19733b2" containerName="revoke-keys"
Nov 02 14:02:00.534844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.534824    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f1a70e-b375-49cc-b456-cf86a19733b2" containerName="revoke-keys"
Nov 02 14:02:00.539401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.539374    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.565574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.565548    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-981903-openshift-default-dockercfg-8nsth\""
Nov 02 14:02:00.586605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.586568    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"]
Nov 02 14:02:00.648622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.648586    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2zq8\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-kube-api-access-t2zq8\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.648818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.648693    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc72acb-c794-4eb1-aff0-f23697179813-istiod-ca-cert\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.648818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.648748    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-socket\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.648950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.648882    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-credential-socket\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.649015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.648985    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-istio-token\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.649087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.649071    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-envoy\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.649141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.649119    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-certs\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.649197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.649160    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc72acb-c794-4eb1-aff0-f23697179813-istio-podinfo\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.649325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.649307    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-data\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.768345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.765938    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-certs\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.768345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.766036    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc72acb-c794-4eb1-aff0-f23697179813-istio-podinfo\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.768345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.766165    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-data\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.768345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.766283    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zq8\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-kube-api-access-t2zq8\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.768345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.766336    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc72acb-c794-4eb1-aff0-f23697179813-istiod-ca-cert\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.768345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.766392    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-socket\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.768345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.766421    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-credential-socket\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.768345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.766464    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-istio-token\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.768345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.766532    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-envoy\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.772793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.771173    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-socket\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.772793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.771263    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-certs\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.772793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.771909    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc72acb-c794-4eb1-aff0-f23697179813-istiod-ca-cert\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.772793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.771947    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-data\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.772793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.772158    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-credential-socket\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.791545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.791441    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc72acb-c794-4eb1-aff0-f23697179813-istio-podinfo\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.798812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.796042    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-envoy\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.802798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.800264    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-istio-token\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.848799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.835958    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zq8\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-kube-api-access-t2zq8\") pod \"e2e-api-b-981903-openshift-default-7dd867666c-vwgbs\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") " pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:00.872812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:00.871251    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:01.301978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:01.301186    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"]
Nov 02 14:02:01.314830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:01.311067    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:01.314830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:01.311189    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:01.314830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:01.311237    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:02.256464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.256422    5034 generic.go:356] "Generic (PLEG): container finished" podID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerID="ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9" exitCode=0
Nov 02 14:02:02.257046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.256524    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" event={"ID":"f34e96f8-c765-4ad6-8cd4-aec1f564ca78","Type":"ContainerDied","Data":"ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9"}
Nov 02 14:02:02.257715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.257689    5034 scope.go:122] "RemoveContainer" containerID="ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9"
Nov 02 14:02:02.260257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.260222    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:02.260382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.260316    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:02.260382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.260363    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:02.261253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.261158    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" event={"ID":"2dc72acb-c794-4eb1-aff0-f23697179813","Type":"ContainerStarted","Data":"5054f5ee21ef0471e0cfbe04a434b98f2ef96dce9d9b94114ae7710aac36ebe5"}
Nov 02 14:02:02.261253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.261215    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" event={"ID":"2dc72acb-c794-4eb1-aff0-f23697179813","Type":"ContainerStarted","Data":"2efdd8ccb3f148ba1ec1ff8bfdcfe0a4677c60f759844061f35b330b52b9f5e7"}
Nov 02 14:02:02.407374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.407097    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" podStartSLOduration=2.407083068 podStartE2EDuration="2.407083068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:02.40286473 +0000 UTC m=+1113.154169169" watchObservedRunningTime="2026-11-02 14:02:02.407083068 +0000 UTC m=+1113.158387539"
Nov 02 14:02:02.878198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.874895    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:02.906927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:02.906894    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:03.291802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:03.287988    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" event={"ID":"f34e96f8-c765-4ad6-8cd4-aec1f564ca78","Type":"ContainerStarted","Data":"ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa"}
Nov 02 14:02:03.291802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:03.289192    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:03.301820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:03.294627    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:03.992802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:03.992085    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:02:03.994933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:03.994903    5034 patch_prober.go:29] interesting pod/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 14:02:03.995087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:03.994965    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 14:02:04.990155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:04.990118    5034 patch_prober.go:29] interesting pod/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 14:02:04.991169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:04.990771    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 14:02:05.034818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:05.034786    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"]
Nov 02 14:02:05.047686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:05.047661    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3346fc4-6kxp4"]
Nov 02 14:02:05.426659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:05.426577    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="311ad3eb-5c96-4aac-a482-649df208e07b" path="/var/lib/kubelet/pods/311ad3eb-5c96-4aac-a482-649df208e07b/volumes"
Nov 02 14:02:05.991803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:05.991102    5034 patch_prober.go:29] interesting pod/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 14:02:05.991803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:05.991173    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 14:02:06.602660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.602622    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"]
Nov 02 14:02:06.606931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.606905    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:06.612998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.612973    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"]
Nov 02 14:02:06.623616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.623587    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-981903\""
Nov 02 14:02:06.637221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.637198    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-981903-dockercfg-qgkpw\""
Nov 02 14:02:06.659917 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.659892    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"]
Nov 02 14:02:06.663391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.663368    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:06.748813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.746177    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxxr\" (UniqueName: \"kubernetes.io/projected/6c508bca-6066-4326-bf21-54460253ad4d-kube-api-access-lkxxr\") pod \"payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn\" (UID: \"6c508bca-6066-4326-bf21-54460253ad4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:06.759135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.759000    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c508bca-6066-4326-bf21-54460253ad4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn\" (UID: \"6c508bca-6066-4326-bf21-54460253ad4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:06.764171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.764142    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd\" (UID: \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:06.809141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.776015    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6wdn\" (UniqueName: \"kubernetes.io/projected/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-kube-api-access-w6wdn\") pod \"payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd\" (UID: \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:06.809141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.793829    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"]
Nov 02 14:02:06.915861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.915407    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxxr\" (UniqueName: \"kubernetes.io/projected/6c508bca-6066-4326-bf21-54460253ad4d-kube-api-access-lkxxr\") pod \"payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn\" (UID: \"6c508bca-6066-4326-bf21-54460253ad4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:06.915861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.915467    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c508bca-6066-4326-bf21-54460253ad4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn\" (UID: \"6c508bca-6066-4326-bf21-54460253ad4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:06.915861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.915609    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd\" (UID: \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:06.915861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.915709    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6wdn\" (UniqueName: \"kubernetes.io/projected/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-kube-api-access-w6wdn\") pod \"payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd\" (UID: \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:06.924845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.923069    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd\" (UID: \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:06.924845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.924542    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c508bca-6066-4326-bf21-54460253ad4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn\" (UID: \"6c508bca-6066-4326-bf21-54460253ad4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:06.974316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.974281    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6wdn\" (UniqueName: \"kubernetes.io/projected/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-kube-api-access-w6wdn\") pod \"payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd\" (UID: \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\") " pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:06.989257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.988767    5034 patch_prober.go:29] interesting pod/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 14:02:06.989257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.988890    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 14:02:06.997079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:06.997049    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxxr\" (UniqueName: \"kubernetes.io/projected/6c508bca-6066-4326-bf21-54460253ad4d-kube-api-access-lkxxr\") pod \"payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn\" (UID: \"6c508bca-6066-4326-bf21-54460253ad4d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:07.139196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:07.139164    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:07.231133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:07.231103    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:07.501148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:07.490695    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"]
Nov 02 14:02:07.513945 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:02:07.513912    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f16e8b8_d839_40fc_95ca_544dee0fb8b5.slice/crio-5848200737e045a4c7a5a33f4e81ee3467bf9baa96064d653f95cfe341998f34 WatchSource:0}: Error finding container 5848200737e045a4c7a5a33f4e81ee3467bf9baa96064d653f95cfe341998f34: Status 404 returned error can't find the container with id 5848200737e045a4c7a5a33f4e81ee3467bf9baa96064d653f95cfe341998f34
Nov 02 14:02:07.606248 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:02:07.606212    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c508bca_6066_4326_bf21_54460253ad4d.slice/crio-f4ccd22506f953417b4d5b77dcb4e799e8785032f800086cc5a90808d2e5660d WatchSource:0}: Error finding container f4ccd22506f953417b4d5b77dcb4e799e8785032f800086cc5a90808d2e5660d: Status 404 returned error can't find the container with id f4ccd22506f953417b4d5b77dcb4e799e8785032f800086cc5a90808d2e5660d
Nov 02 14:02:07.606455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:07.606429    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"]
Nov 02 14:02:07.703591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:07.703567    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:02:08.000825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:07.995888    5034 patch_prober.go:29] interesting pod/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Nov 02 14:02:08.000825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:07.995948    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Nov 02 14:02:08.361516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.361419    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd" event={"ID":"2f16e8b8-d839-40fc-95ca-544dee0fb8b5","Type":"ContainerStarted","Data":"afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b"}
Nov 02 14:02:08.361516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.361491    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd" event={"ID":"2f16e8b8-d839-40fc-95ca-544dee0fb8b5","Type":"ContainerStarted","Data":"5848200737e045a4c7a5a33f4e81ee3467bf9baa96064d653f95cfe341998f34"}
Nov 02 14:02:08.361750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.361527    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:08.364213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.364187    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn" event={"ID":"6c508bca-6066-4326-bf21-54460253ad4d","Type":"ContainerStarted","Data":"fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6"}
Nov 02 14:02:08.364305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.364229    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn" event={"ID":"6c508bca-6066-4326-bf21-54460253ad4d","Type":"ContainerStarted","Data":"f4ccd22506f953417b4d5b77dcb4e799e8785032f800086cc5a90808d2e5660d"}
Nov 02 14:02:08.364350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.364334    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:08.417512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.417449    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd" podStartSLOduration=2.417435807 podStartE2EDuration="2.417435807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:08.411050241 +0000 UTC m=+1119.162354680" watchObservedRunningTime="2026-11-02 14:02:08.417435807 +0000 UTC m=+1119.168740258"
Nov 02 14:02:08.461577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.461175    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn" podStartSLOduration=2.461160699 podStartE2EDuration="2.461160699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:08.455648572 +0000 UTC m=+1119.206953001" watchObservedRunningTime="2026-11-02 14:02:08.461160699 +0000 UTC m=+1119.212465168"
Nov 02 14:02:08.618611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.618521    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"]
Nov 02 14:02:08.622892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.622868    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:08.641841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.641771    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-5e9a9d\""
Nov 02 14:02:08.645895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.645868    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"]
Nov 02 14:02:08.806835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.799264    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9pl\" (UniqueName: \"kubernetes.io/projected/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-kube-api-access-6v9pl\") pod \"payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z\" (UID: \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:08.806835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.799570    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z\" (UID: \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:08.940817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.938453    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z\" (UID: \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:08.940817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.938628    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9pl\" (UniqueName: \"kubernetes.io/projected/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-kube-api-access-6v9pl\") pod \"payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z\" (UID: \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:08.958831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.956233    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z\" (UID: \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:08.996356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:08.995498    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9pl\" (UniqueName: \"kubernetes.io/projected/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-kube-api-access-6v9pl\") pod \"payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z\" (UID: \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\") " pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:09.011112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:09.009449    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:02:09.012522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:09.012498    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:02:09.239293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:09.239261    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:09.554665 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:02:09.554638    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7e0a7f_60c5_41ee_9c67_8b019de192f9.slice/crio-4e63888a9bdbbeac81ede15d83748aaa30986c93a0139757c82596fd2e4a469c WatchSource:0}: Error finding container 4e63888a9bdbbeac81ede15d83748aaa30986c93a0139757c82596fd2e4a469c: Status 404 returned error can't find the container with id 4e63888a9bdbbeac81ede15d83748aaa30986c93a0139757c82596fd2e4a469c
Nov 02 14:02:09.556233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:09.556206    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"]
Nov 02 14:02:10.401847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:10.401717    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z" event={"ID":"4c7e0a7f-60c5-41ee-9c67-8b019de192f9","Type":"ContainerStarted","Data":"ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218"}
Nov 02 14:02:10.402324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:10.401847    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z" event={"ID":"4c7e0a7f-60c5-41ee-9c67-8b019de192f9","Type":"ContainerStarted","Data":"4e63888a9bdbbeac81ede15d83748aaa30986c93a0139757c82596fd2e4a469c"}
Nov 02 14:02:10.402324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:10.402210    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:10.450294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:10.450233    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z" podStartSLOduration=2.450217358 podStartE2EDuration="2.450217358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:10.447902247 +0000 UTC m=+1121.199206721" watchObservedRunningTime="2026-11-02 14:02:10.450217358 +0000 UTC m=+1121.201521797"
Nov 02 14:02:15.045800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:15.036880    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"]
Nov 02 14:02:15.053844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:15.053025    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d660a456-9jc8l"]
Nov 02 14:02:15.429533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:15.429435    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e10c7f-4197-4922-b12c-fd7bbb573aed" path="/var/lib/kubelet/pods/f8e10c7f-4197-4922-b12c-fd7bbb573aed/volumes"
Nov 02 14:02:17.361903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:17.361857    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:02:17.392818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:17.385063    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:02:17.392818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:17.386110    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:02:17.482325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:17.481366    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:02:17.886534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:17.886486    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:02:17.896808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:17.893587    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:02:17.920198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:17.920157    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:02:18.429825 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:02:18.429761    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:02:19.399197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:19.399168    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:02:19.399373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:19.399230    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:02:20.034022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.033991    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"]
Nov 02 14:02:20.047040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.047006    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a57390c7-mkggv"]
Nov 02 14:02:20.368552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.368473    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"]
Nov 02 14:02:20.373128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.373103    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.394509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.394481    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"]
Nov 02 14:02:20.400608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.400571    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:02:20.400864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.400623    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:02:20.401006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.400814    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:02:20.401161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.401148    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:02:20.453044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.453008    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ac7333-4gslh\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.453400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.453371    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ac7333-4gslh\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.453526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.453411    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwl6\" (UniqueName: \"kubernetes.io/projected/4eca4eb5-319a-4629-8586-fd69b86c64c7-kube-api-access-xjwl6\") pod \"maas-api-revoke-sub-35ac7333-4gslh\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.554428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.554397    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ac7333-4gslh\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.554636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.554617    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ac7333-4gslh\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.554753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.554649    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwl6\" (UniqueName: \"kubernetes.io/projected/4eca4eb5-319a-4629-8586-fd69b86c64c7-kube-api-access-xjwl6\") pod \"maas-api-revoke-sub-35ac7333-4gslh\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.555248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.555220    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35ac7333-4gslh\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.557185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.557160    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35ac7333-4gslh\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.595551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.595521    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwl6\" (UniqueName: \"kubernetes.io/projected/4eca4eb5-319a-4629-8586-fd69b86c64c7-kube-api-access-xjwl6\") pod \"maas-api-revoke-sub-35ac7333-4gslh\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:20.690014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:20.689928    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:21.013243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:21.013214    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"]
Nov 02 14:02:21.398956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:21.398921    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c92f72-a156-4100-b7e7-28a41e871bb5" path="/var/lib/kubelet/pods/a9c92f72-a156-4100-b7e7-28a41e871bb5/volumes"
Nov 02 14:02:21.423769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:21.423747    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:02:21.634574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:21.634474    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh" event={"ID":"4eca4eb5-319a-4629-8586-fd69b86c64c7","Type":"ContainerStarted","Data":"26c3a6f41de3b4300ad8a70d6a32b2dc5bbd3117840ca734ee2562bd2cd7c0c9"}
Nov 02 14:02:21.634746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:21.634594    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh" event={"ID":"4eca4eb5-319a-4629-8586-fd69b86c64c7","Type":"ContainerStarted","Data":"4fb0962e2b39c4ea3da5e252b75e0b3561770a5f98d33229847225929f24ec62"}
Nov 02 14:02:22.644835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:22.644797    5034 generic.go:356] "Generic (PLEG): container finished" podID="4eca4eb5-319a-4629-8586-fd69b86c64c7" containerID="26c3a6f41de3b4300ad8a70d6a32b2dc5bbd3117840ca734ee2562bd2cd7c0c9" exitCode=0
Nov 02 14:02:22.645302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:22.644882    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh" event={"ID":"4eca4eb5-319a-4629-8586-fd69b86c64c7","Type":"ContainerDied","Data":"26c3a6f41de3b4300ad8a70d6a32b2dc5bbd3117840ca734ee2562bd2cd7c0c9"}
Nov 02 14:02:23.833519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:23.833494    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:23.934473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:23.934399    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-openshift-ca\") pod \"4eca4eb5-319a-4629-8586-fd69b86c64c7\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") "
Nov 02 14:02:23.934622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:23.934475    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eca4eb5-319a-4629-8586-fd69b86c64c7-kube-api-access-xjwl6\" (UniqueName: \"kubernetes.io/projected/4eca4eb5-319a-4629-8586-fd69b86c64c7-kube-api-access-xjwl6\") pod \"4eca4eb5-319a-4629-8586-fd69b86c64c7\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") "
Nov 02 14:02:23.934850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:23.934821    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-openshift-ca" pod "4eca4eb5-319a-4629-8586-fd69b86c64c7" (UID: "4eca4eb5-319a-4629-8586-fd69b86c64c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:02:23.934946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:23.934835    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-xks-ca\") pod \"4eca4eb5-319a-4629-8586-fd69b86c64c7\" (UID: \"4eca4eb5-319a-4629-8586-fd69b86c64c7\") "
Nov 02 14:02:23.935657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:23.935632    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:23.936888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:23.936860    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eca4eb5-319a-4629-8586-fd69b86c64c7-kube-api-access-xjwl6" pod "4eca4eb5-319a-4629-8586-fd69b86c64c7" (UID: "4eca4eb5-319a-4629-8586-fd69b86c64c7"). InnerVolumeSpecName "kube-api-access-xjwl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:02:23.937012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:23.936987    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-xks-ca" pod "4eca4eb5-319a-4629-8586-fd69b86c64c7" (UID: "4eca4eb5-319a-4629-8586-fd69b86c64c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:02:24.037685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:24.037649    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eca4eb5-319a-4629-8586-fd69b86c64c7-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:24.037685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:24.037680    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjwl6\" (UniqueName: \"kubernetes.io/projected/4eca4eb5-319a-4629-8586-fd69b86c64c7-kube-api-access-xjwl6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:24.664061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:24.664031    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"
Nov 02 14:02:24.664247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:24.664071    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh" event={"ID":"4eca4eb5-319a-4629-8586-fd69b86c64c7","Type":"ContainerDied","Data":"4fb0962e2b39c4ea3da5e252b75e0b3561770a5f98d33229847225929f24ec62"}
Nov 02 14:02:24.664247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:24.664123    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb0962e2b39c4ea3da5e252b75e0b3561770a5f98d33229847225929f24ec62"
Nov 02 14:02:27.659857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.659817    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7036-ecaae5"]
Nov 02 14:02:27.660732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.660708    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4eca4eb5-319a-4629-8586-fd69b86c64c7" containerName="revoke-keys"
Nov 02 14:02:27.660732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.660733    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eca4eb5-319a-4629-8586-fd69b86c64c7" containerName="revoke-keys"
Nov 02 14:02:27.661023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.661007    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eca4eb5-319a-4629-8586-fd69b86c64c7" containerName="revoke-keys"
Nov 02 14:02:27.664940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.664919    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:27.681897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.681876    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7jzh4\""
Nov 02 14:02:27.685133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.685110    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7036-ecaae5"]
Nov 02 14:02:27.809741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.809714    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqxbm\" (UniqueName: \"kubernetes.io/projected/09622186-ca1e-42ee-a4ad-6469e0be59ea-kube-api-access-nqxbm\") pod \"test-curl-7036-ecaae5\" (UID: \"09622186-ca1e-42ee-a4ad-6469e0be59ea\") " pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:27.911348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.911254    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqxbm\" (UniqueName: \"kubernetes.io/projected/09622186-ca1e-42ee-a4ad-6469e0be59ea-kube-api-access-nqxbm\") pod \"test-curl-7036-ecaae5\" (UID: \"09622186-ca1e-42ee-a4ad-6469e0be59ea\") " pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:27.937018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.936992    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqxbm\" (UniqueName: \"kubernetes.io/projected/09622186-ca1e-42ee-a4ad-6469e0be59ea-kube-api-access-nqxbm\") pod \"test-curl-7036-ecaae5\" (UID: \"09622186-ca1e-42ee-a4ad-6469e0be59ea\") " pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:27.980175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:27.980151    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:28.286875 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:02:28.286844    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09622186_ca1e_42ee_a4ad_6469e0be59ea.slice/crio-d00519dc18910de58a61af82a6c42fe14ef546cf3b47065b77ddd9b3f1862bbd WatchSource:0}: Error finding container d00519dc18910de58a61af82a6c42fe14ef546cf3b47065b77ddd9b3f1862bbd: Status 404 returned error can't find the container with id d00519dc18910de58a61af82a6c42fe14ef546cf3b47065b77ddd9b3f1862bbd
Nov 02 14:02:28.289520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:28.289496    5034 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 14:02:28.289982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:28.289957    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7036-ecaae5"]
Nov 02 14:02:28.701346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:28.701249    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7036-ecaae5" event={"ID":"09622186-ca1e-42ee-a4ad-6469e0be59ea","Type":"ContainerStarted","Data":"d00519dc18910de58a61af82a6c42fe14ef546cf3b47065b77ddd9b3f1862bbd"}
Nov 02 14:02:30.722767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:30.722730    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7036-ecaae5" event={"ID":"09622186-ca1e-42ee-a4ad-6469e0be59ea","Type":"ContainerStarted","Data":"d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c"}
Nov 02 14:02:30.767713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:30.767668    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7036-ecaae5" podStartSLOduration=1.4947092419999999 podStartE2EDuration="3.767654713s" totalImagesPullingTime="2.272945471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:30.763579831 +0000 UTC m=+1141.514884269" watchObservedRunningTime="2026-11-02 14:02:30.767654713 +0000 UTC m=+1141.518959151"
Nov 02 14:02:31.506439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.506093    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"]
Nov 02 14:02:31.511598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.511571    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.545484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.545450    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"]
Nov 02 14:02:31.553820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.553751    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-584171-openshift-default-dockercfg-vr87q\""
Nov 02 14:02:31.569060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.569029    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-envoy\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.569204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.569081    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6328bd-5689-4d4e-9fa0-3ab417072485-istiod-ca-cert\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.569274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.569214    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-token\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.569364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.569346    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-data\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.569468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.569451    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvcvz\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-kube-api-access-zvcvz\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.569556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.569518    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-socket\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.569618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.569550    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-podinfo\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.569681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.569661    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-credential-socket\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.569736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.569714    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-certs\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.688805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.687975    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-token\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.688805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.688146    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-data\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.688805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.688261    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvcvz\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-kube-api-access-zvcvz\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.688805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.688333    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-socket\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.688805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.688369    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-podinfo\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.688805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.688420    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-credential-socket\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.688805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.688474    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-certs\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.688805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.688536    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-envoy\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.688805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.688574    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6328bd-5689-4d4e-9fa0-3ab417072485-istiod-ca-cert\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.689567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.689451    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6328bd-5689-4d4e-9fa0-3ab417072485-istiod-ca-cert\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.804853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.798267    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-socket\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.804853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.798533    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-certs\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.804853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.798853    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-credential-socket\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.804853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.803971    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7036-ecaae5"]
Nov 02 14:02:31.804853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.804283    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-data\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.920880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.898133    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7036-ecaae5"]
Nov 02 14:02:31.920880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.908743    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvcvz\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-kube-api-access-zvcvz\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.920880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.909004    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-envoy\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.920880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.912459    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-token\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:31.920880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:31.913251    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-podinfo\") pod \"e2e-api-a-584171-openshift-default-57cdf6487c-tln9z\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") " pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:32.144208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.144132    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:32.387824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.387788    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"]
Nov 02 14:02:32.388917 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.388877    5034 cpu_manager.go:414] "containerMap: removing container" podUID="09622186-ca1e-42ee-a4ad-6469e0be59ea" containerName="test-curl-7036-ecaae5"
Nov 02 14:02:32.388917 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.388913    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09622186-ca1e-42ee-a4ad-6469e0be59ea" containerName="test-curl-7036-ecaae5"
Nov 02 14:02:32.389176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.389162    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="09622186-ca1e-42ee-a4ad-6469e0be59ea" containerName="test-curl-7036-ecaae5"
Nov 02 14:02:32.396398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.396323    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.417310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.415302    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"]
Nov 02 14:02:32.419600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.419258    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:02:32.419600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.419360    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:02:32.419600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.419402    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:02:32.420106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.420085    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d776122a-qxbw9\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.420241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.420223    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qrcd\" (UniqueName: \"kubernetes.io/projected/8be4e967-c791-491b-b9cf-7375cfd4e0bd-kube-api-access-2qrcd\") pod \"maas-api-revoke-keys-e2e-mt-d776122a-qxbw9\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.420438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.420422    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d776122a-qxbw9\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.426346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.426315    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:02:32.492232 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:02:32.492201    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6328bd_5689_4d4e_9fa0_3ab417072485.slice/crio-c8660fbf340057f54f8c45d49961f61d64077a39d86821b41a31e2700b4e6d1d WatchSource:0}: Error finding container c8660fbf340057f54f8c45d49961f61d64077a39d86821b41a31e2700b4e6d1d: Status 404 returned error can't find the container with id c8660fbf340057f54f8c45d49961f61d64077a39d86821b41a31e2700b4e6d1d
Nov 02 14:02:32.495171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.495127    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:32.495479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.495217    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:32.495479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.495267    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:32.497730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.497700    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"]
Nov 02 14:02:32.522161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.522136    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qrcd\" (UniqueName: \"kubernetes.io/projected/8be4e967-c791-491b-b9cf-7375cfd4e0bd-kube-api-access-2qrcd\") pod \"maas-api-revoke-keys-e2e-mt-d776122a-qxbw9\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.522354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.522334    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d776122a-qxbw9\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.522497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.522480    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d776122a-qxbw9\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.523145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.523128    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d776122a-qxbw9\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.524836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.524770    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d776122a-qxbw9\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.548657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.548637    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qrcd\" (UniqueName: \"kubernetes.io/projected/8be4e967-c791-491b-b9cf-7375cfd4e0bd-kube-api-access-2qrcd\") pod \"maas-api-revoke-keys-e2e-mt-d776122a-qxbw9\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.719906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.717530    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:32.955368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.955329    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" event={"ID":"dd6328bd-5689-4d4e-9fa0-3ab417072485","Type":"ContainerStarted","Data":"e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597"}
Nov 02 14:02:32.955850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.955398    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" event={"ID":"dd6328bd-5689-4d4e-9fa0-3ab417072485","Type":"ContainerStarted","Data":"c8660fbf340057f54f8c45d49961f61d64077a39d86821b41a31e2700b4e6d1d"}
Nov 02 14:02:32.955850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:32.955439    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7036-ecaae5" podUID="09622186-ca1e-42ee-a4ad-6469e0be59ea" containerName="test-curl-7036-ecaae5" containerID="cri-o://d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c" gracePeriod=2
Nov 02 14:02:32.970911 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:02:32.970826    5034 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7036-ecaae5\" is forbidden: User \"system:node:ip-10-0-139-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-26.ec2.internal' and this object" podUID="09622186-ca1e-42ee-a4ad-6469e0be59ea" pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:33.019549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:33.019501    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" podStartSLOduration=2.019487627 podStartE2EDuration="2.019487627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:33.018579954 +0000 UTC m=+1143.769884393" watchObservedRunningTime="2026-11-02 14:02:33.019487627 +0000 UTC m=+1143.770792064"
Nov 02 14:02:33.033587 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:02:33.033557    5034 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7036-ecaae5\" is forbidden: User \"system:node:ip-10-0-139-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-26.ec2.internal' and this object" podUID="09622186-ca1e-42ee-a4ad-6469e0be59ea" pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:33.057387 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:02:33.057359    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be4e967_c791_491b_b9cf_7375cfd4e0bd.slice/crio-448f07b97e24c2d7de7a18cb75a8dcb245acf3d0d5a6a9c7d81ddda1f0886905 WatchSource:0}: Error finding container 448f07b97e24c2d7de7a18cb75a8dcb245acf3d0d5a6a9c7d81ddda1f0886905: Status 404 returned error can't find the container with id 448f07b97e24c2d7de7a18cb75a8dcb245acf3d0d5a6a9c7d81ddda1f0886905
Nov 02 14:02:33.059515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:33.059496    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"]
Nov 02 14:02:33.144843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:33.144818    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:33.150589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:33.150569    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:33.165497 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:02:33.165469    5034 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7036-ecaae5\" is forbidden: User \"system:node:ip-10-0-139-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-26.ec2.internal' and this object" podUID="09622186-ca1e-42ee-a4ad-6469e0be59ea" pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:33.965646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:33.965610    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9" event={"ID":"8be4e967-c791-491b-b9cf-7375cfd4e0bd","Type":"ContainerStarted","Data":"578c7eef180d9f7f8258fdbdf85980ab333af0c5d66bba1555f9760958bd23e2"}
Nov 02 14:02:33.966073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:33.965660    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9" event={"ID":"8be4e967-c791-491b-b9cf-7375cfd4e0bd","Type":"ContainerStarted","Data":"448f07b97e24c2d7de7a18cb75a8dcb245acf3d0d5a6a9c7d81ddda1f0886905"}
Nov 02 14:02:33.966138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:33.966092    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:33.967069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:33.967045    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:02:34.013104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:34.013052    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9" podStartSLOduration=2.013036147 podStartE2EDuration="2.013036147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:34.01053957 +0000 UTC m=+1144.761844009" watchObservedRunningTime="2026-11-02 14:02:34.013036147 +0000 UTC m=+1144.764340585"
Nov 02 14:02:34.450099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:34.450068    5034 scope.go:122] "RemoveContainer" containerID="6703efcad27e1855599e34b07ce40623d8c94e207dc107b073e2116db533efd7"
Nov 02 14:02:34.461063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:34.461038    5034 scope.go:122] "RemoveContainer" containerID="4e446fa3ea09298b0c8ff047207feaa4af8d46f94968edddfee0a0d4198f3911"
Nov 02 14:02:34.471826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:34.471765    5034 scope.go:122] "RemoveContainer" containerID="270bc9155027cce30ded72f8c4b9891985dc888f7a6e8a0e19fb028be3961fcf"
Nov 02 14:02:34.483670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:34.483650    5034 scope.go:122] "RemoveContainer" containerID="7d093c6eb77a23442e6aec5b250bcfd2e48cfa8bb86d12988ff78b95e6017a0b"
Nov 02 14:02:34.977633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:34.977608    5034 generic.go:356] "Generic (PLEG): container finished" podID="8be4e967-c791-491b-b9cf-7375cfd4e0bd" containerID="578c7eef180d9f7f8258fdbdf85980ab333af0c5d66bba1555f9760958bd23e2" exitCode=0
Nov 02 14:02:34.977958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:34.977696    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9" event={"ID":"8be4e967-c791-491b-b9cf-7375cfd4e0bd","Type":"ContainerDied","Data":"578c7eef180d9f7f8258fdbdf85980ab333af0c5d66bba1555f9760958bd23e2"}
Nov 02 14:02:35.269481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:35.269456    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:35.383105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:35.383077    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09622186-ca1e-42ee-a4ad-6469e0be59ea-kube-api-access-nqxbm\" (UniqueName: \"kubernetes.io/projected/09622186-ca1e-42ee-a4ad-6469e0be59ea-kube-api-access-nqxbm\") pod \"09622186-ca1e-42ee-a4ad-6469e0be59ea\" (UID: \"09622186-ca1e-42ee-a4ad-6469e0be59ea\") "
Nov 02 14:02:35.385910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:35.385875    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09622186-ca1e-42ee-a4ad-6469e0be59ea-kube-api-access-nqxbm" pod "09622186-ca1e-42ee-a4ad-6469e0be59ea" (UID: "09622186-ca1e-42ee-a4ad-6469e0be59ea"). InnerVolumeSpecName "kube-api-access-nqxbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:02:35.399591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:35.399562    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09622186-ca1e-42ee-a4ad-6469e0be59ea" path="/var/lib/kubelet/pods/09622186-ca1e-42ee-a4ad-6469e0be59ea/volumes"
Nov 02 14:02:35.485556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:35.485512    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqxbm\" (UniqueName: \"kubernetes.io/projected/09622186-ca1e-42ee-a4ad-6469e0be59ea-kube-api-access-nqxbm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:35.988340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:35.988304    5034 generic.go:356] "Generic (PLEG): container finished" podID="09622186-ca1e-42ee-a4ad-6469e0be59ea" containerID="d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c" exitCode=137
Nov 02 14:02:35.988870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:35.988349    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-ecaae5"
Nov 02 14:02:35.988870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:35.988409    5034 scope.go:122] "RemoveContainer" containerID="d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c"
Nov 02 14:02:36.011273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.011247    5034 scope.go:122] "RemoveContainer" containerID="d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c"
Nov 02 14:02:36.011656 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:02:36.011629    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c\": container with ID starting with d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c not found: ID does not exist" containerID="d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c"
Nov 02 14:02:36.011732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.011671    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c"} err="failed to get container status \"d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c\": rpc error: code = NotFound desc = could not find container \"d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c\": container with ID starting with d44ecf0bba8208975b79de6b7ffd50c5fd23df6fa28213de1d27e1cc2ee9653c not found: ID does not exist"
Nov 02 14:02:36.178409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.178377    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:36.196405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.196371    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be4e967-c791-491b-b9cf-7375cfd4e0bd-kube-api-access-2qrcd\" (UniqueName: \"kubernetes.io/projected/8be4e967-c791-491b-b9cf-7375cfd4e0bd-kube-api-access-2qrcd\") pod \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") "
Nov 02 14:02:36.196537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.196445    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-xks-ca\") pod \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") "
Nov 02 14:02:36.196537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.196488    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-openshift-ca\") pod \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\" (UID: \"8be4e967-c791-491b-b9cf-7375cfd4e0bd\") "
Nov 02 14:02:36.196936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.196907    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-openshift-ca" pod "8be4e967-c791-491b-b9cf-7375cfd4e0bd" (UID: "8be4e967-c791-491b-b9cf-7375cfd4e0bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:02:36.197421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.197398    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:36.198826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.198767    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be4e967-c791-491b-b9cf-7375cfd4e0bd-kube-api-access-2qrcd" pod "8be4e967-c791-491b-b9cf-7375cfd4e0bd" (UID: "8be4e967-c791-491b-b9cf-7375cfd4e0bd"). InnerVolumeSpecName "kube-api-access-2qrcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:02:36.199072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.199047    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-xks-ca" pod "8be4e967-c791-491b-b9cf-7375cfd4e0bd" (UID: "8be4e967-c791-491b-b9cf-7375cfd4e0bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:02:36.298529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.298495    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qrcd\" (UniqueName: \"kubernetes.io/projected/8be4e967-c791-491b-b9cf-7375cfd4e0bd-kube-api-access-2qrcd\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:36.298529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.298527    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be4e967-c791-491b-b9cf-7375cfd4e0bd-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:36.999735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.999701    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"
Nov 02 14:02:36.999735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.999728    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9" event={"ID":"8be4e967-c791-491b-b9cf-7375cfd4e0bd","Type":"ContainerDied","Data":"448f07b97e24c2d7de7a18cb75a8dcb245acf3d0d5a6a9c7d81ddda1f0886905"}
Nov 02 14:02:37.000281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:36.999793    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="448f07b97e24c2d7de7a18cb75a8dcb245acf3d0d5a6a9c7d81ddda1f0886905"
Nov 02 14:02:37.681803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.678254    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"]
Nov 02 14:02:37.681803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.679477    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8be4e967-c791-491b-b9cf-7375cfd4e0bd" containerName="revoke-keys"
Nov 02 14:02:37.681803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.679500    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be4e967-c791-491b-b9cf-7375cfd4e0bd" containerName="revoke-keys"
Nov 02 14:02:37.681803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.679749    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be4e967-c791-491b-b9cf-7375cfd4e0bd" containerName="revoke-keys"
Nov 02 14:02:37.797943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.797912    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:37.827156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.827122    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"]
Nov 02 14:02:37.851959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.851927    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd627ac5-e1ee-48aa-af50-de344e035e73-plugins-config-volume\") pod \"payload-processing-e2e-api-a-584171-688bc6999-fk9kg\" (UID: \"fd627ac5-e1ee-48aa-af50-de344e035e73\") " pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:37.852441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.852420    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwch\" (UniqueName: \"kubernetes.io/projected/fd627ac5-e1ee-48aa-af50-de344e035e73-kube-api-access-njwch\") pod \"payload-processing-e2e-api-a-584171-688bc6999-fk9kg\" (UID: \"fd627ac5-e1ee-48aa-af50-de344e035e73\") " pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:37.911831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.892887    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-584171\""
Nov 02 14:02:37.911831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.892970    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-584171-dockercfg-m5v8m\""
Nov 02 14:02:37.954644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.954551    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwch\" (UniqueName: \"kubernetes.io/projected/fd627ac5-e1ee-48aa-af50-de344e035e73-kube-api-access-njwch\") pod \"payload-processing-e2e-api-a-584171-688bc6999-fk9kg\" (UID: \"fd627ac5-e1ee-48aa-af50-de344e035e73\") " pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:37.954868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.954723    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd627ac5-e1ee-48aa-af50-de344e035e73-plugins-config-volume\") pod \"payload-processing-e2e-api-a-584171-688bc6999-fk9kg\" (UID: \"fd627ac5-e1ee-48aa-af50-de344e035e73\") " pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:37.955744 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.955716    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd627ac5-e1ee-48aa-af50-de344e035e73-plugins-config-volume\") pod \"payload-processing-e2e-api-a-584171-688bc6999-fk9kg\" (UID: \"fd627ac5-e1ee-48aa-af50-de344e035e73\") " pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:37.982503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:37.982472    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwch\" (UniqueName: \"kubernetes.io/projected/fd627ac5-e1ee-48aa-af50-de344e035e73-kube-api-access-njwch\") pod \"payload-processing-e2e-api-a-584171-688bc6999-fk9kg\" (UID: \"fd627ac5-e1ee-48aa-af50-de344e035e73\") " pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:38.243243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:38.243207    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:39.007541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:39.007379    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"]
Nov 02 14:02:39.053834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:39.053796    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg" event={"ID":"fd627ac5-e1ee-48aa-af50-de344e035e73","Type":"ContainerStarted","Data":"150d879d1667f69ce1b066f264be061f73d33a9a6c484c5dd0cb3b9bba168fb2"}
Nov 02 14:02:40.064358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:40.064317    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg" event={"ID":"fd627ac5-e1ee-48aa-af50-de344e035e73","Type":"ContainerStarted","Data":"89656b8691ba984de993a4c2d7700899d1b8e96c1d6497f8399af9626c34f89d"}
Nov 02 14:02:40.064867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:40.064431    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:40.111003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:40.110955    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg" podStartSLOduration=3.110942904 podStartE2EDuration="3.110942904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:40.109091839 +0000 UTC m=+1150.860396283" watchObservedRunningTime="2026-11-02 14:02:40.110942904 +0000 UTC m=+1150.862247341"
Nov 02 14:02:43.807249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:43.807207    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"]
Nov 02 14:02:46.311702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.311663    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"]
Nov 02 14:02:46.333002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.332967    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.341657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.336983    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"]
Nov 02 14:02:46.388700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.388149    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:02:46.388700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.388279    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:02:46.399153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.398242    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:02:46.406865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.406212    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:02:46.576894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.562846    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c9af55-pnb9x\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.576894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.563017    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj7wr\" (UniqueName: \"kubernetes.io/projected/06467082-aef9-4454-9f96-c8f21c174b9b-kube-api-access-tj7wr\") pod \"maas-api-revoke-sub-b0c9af55-pnb9x\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.576894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.563298    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c9af55-pnb9x\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.664536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.664500    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c9af55-pnb9x\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.664685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.664573    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c9af55-pnb9x\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.664739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.664685    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj7wr\" (UniqueName: \"kubernetes.io/projected/06467082-aef9-4454-9f96-c8f21c174b9b-kube-api-access-tj7wr\") pod \"maas-api-revoke-sub-b0c9af55-pnb9x\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.665296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.665271    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c9af55-pnb9x\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.666862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.666841    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c9af55-pnb9x\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.693561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.693535    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj7wr\" (UniqueName: \"kubernetes.io/projected/06467082-aef9-4454-9f96-c8f21c174b9b-kube-api-access-tj7wr\") pod \"maas-api-revoke-sub-b0c9af55-pnb9x\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:46.707447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:46.707419    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:47.079440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:47.077866    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"]
Nov 02 14:02:47.199847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:47.199810    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x" event={"ID":"06467082-aef9-4454-9f96-c8f21c174b9b","Type":"ContainerStarted","Data":"b00d513e6dc5a4678d5ada42e2f2c88f2ae06999e4faafeaf54de3eca6f757c9"}
Nov 02 14:02:48.215077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:48.215042    5034 generic.go:356] "Generic (PLEG): container finished" podID="06467082-aef9-4454-9f96-c8f21c174b9b" containerID="cff1ad560af18b84e9337c4985a155d1a0648a378a54aa10006cc6b67143f8d0" exitCode=0
Nov 02 14:02:48.215552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:48.215125    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x" event={"ID":"06467082-aef9-4454-9f96-c8f21c174b9b","Type":"ContainerDied","Data":"cff1ad560af18b84e9337c4985a155d1a0648a378a54aa10006cc6b67143f8d0"}
Nov 02 14:02:48.416473 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:02:48.416375    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:02:48.632221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:48.629912    5034 patch_prober.go:29] interesting pod/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": read tcp 10.132.0.2:34102->10.132.0.139:15021: read: connection reset by peer" start-of-body=
Nov 02 14:02:48.632221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:48.629974    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": read tcp 10.132.0.2:34102->10.132.0.139:15021: read: connection reset by peer"
Nov 02 14:02:48.808286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:48.808200    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf" podUID="867a30da-620a-4c25-a569-c300147d5264" containerName="payload-pre-processing" containerID="cri-o://b37a75f8f4dc80a9104ef782db157a255c0f750a11605d26d8c24ee00a6bc6de" gracePeriod=25
Nov 02 14:02:49.224877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.224797    5034 generic.go:356] "Generic (PLEG): container finished" podID="2dc72acb-c794-4eb1-aff0-f23697179813" containerID="5054f5ee21ef0471e0cfbe04a434b98f2ef96dce9d9b94114ae7710aac36ebe5" exitCode=0
Nov 02 14:02:49.224877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.224814    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" event={"ID":"2dc72acb-c794-4eb1-aff0-f23697179813","Type":"ContainerDied","Data":"5054f5ee21ef0471e0cfbe04a434b98f2ef96dce9d9b94114ae7710aac36ebe5"}
Nov 02 14:02:49.225407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.225374    5034 scope.go:122] "RemoveContainer" containerID="5054f5ee21ef0471e0cfbe04a434b98f2ef96dce9d9b94114ae7710aac36ebe5"
Nov 02 14:02:49.227141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.227112    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:49.227247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.227179    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:49.227247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.227213    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:02:49.457939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.456383    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:49.530342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.530298    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-xks-ca\") pod \"06467082-aef9-4454-9f96-c8f21c174b9b\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") "
Nov 02 14:02:49.530536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.530399    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-openshift-ca\") pod \"06467082-aef9-4454-9f96-c8f21c174b9b\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") "
Nov 02 14:02:49.530536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.530444    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06467082-aef9-4454-9f96-c8f21c174b9b-kube-api-access-tj7wr\" (UniqueName: \"kubernetes.io/projected/06467082-aef9-4454-9f96-c8f21c174b9b-kube-api-access-tj7wr\") pod \"06467082-aef9-4454-9f96-c8f21c174b9b\" (UID: \"06467082-aef9-4454-9f96-c8f21c174b9b\") "
Nov 02 14:02:49.534374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.534342    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-xks-ca" pod "06467082-aef9-4454-9f96-c8f21c174b9b" (UID: "06467082-aef9-4454-9f96-c8f21c174b9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:02:49.534514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.534466    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-openshift-ca" pod "06467082-aef9-4454-9f96-c8f21c174b9b" (UID: "06467082-aef9-4454-9f96-c8f21c174b9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:02:49.540306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.540277    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06467082-aef9-4454-9f96-c8f21c174b9b-kube-api-access-tj7wr" pod "06467082-aef9-4454-9f96-c8f21c174b9b" (UID: "06467082-aef9-4454-9f96-c8f21c174b9b"). InnerVolumeSpecName "kube-api-access-tj7wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:02:49.632999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.632962    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:49.632999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.632993    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06467082-aef9-4454-9f96-c8f21c174b9b-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:49.633180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:49.633008    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj7wr\" (UniqueName: \"kubernetes.io/projected/06467082-aef9-4454-9f96-c8f21c174b9b-kube-api-access-tj7wr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:02:50.236456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:50.236419    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" event={"ID":"2dc72acb-c794-4eb1-aff0-f23697179813","Type":"ContainerStarted","Data":"eca2a0d221d47b45122059b4b51ef7c9fc3861755c5ce5c9a68fc15750449e39"}
Nov 02 14:02:50.238011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:50.237981    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"
Nov 02 14:02:50.238011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:50.238006    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x" event={"ID":"06467082-aef9-4454-9f96-c8f21c174b9b","Type":"ContainerDied","Data":"b00d513e6dc5a4678d5ada42e2f2c88f2ae06999e4faafeaf54de3eca6f757c9"}
Nov 02 14:02:50.238200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:50.238043    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b00d513e6dc5a4678d5ada42e2f2c88f2ae06999e4faafeaf54de3eca6f757c9"
Nov 02 14:02:50.450673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:50.450626    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf" podUID="867a30da-620a-4c25-a569-c300147d5264" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:02:50.895288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:50.895125    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:50.895497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:50.895410    5034 patch_prober.go:29] interesting pod/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:02:50.895497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:50.895463    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 14:02:51.094800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:51.094597    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:02:51.879157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:51.879120    5034 patch_prober.go:29] interesting pod/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:02:51.879754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:51.879720    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 14:02:52.888638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:52.888585    5034 patch_prober.go:29] interesting pod/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:02:52.890079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:52.890044    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 14:02:53.901275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:53.885851    5034 patch_prober.go:29] interesting pod/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:02:53.901275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:53.885920    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 14:02:54.882466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:54.882424    5034 patch_prober.go:29] interesting pod/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Nov 02 14:02:54.883654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:54.883261    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Nov 02 14:02:55.167063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:55.166973    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"]
Nov 02 14:02:55.205448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:55.205405    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"]
Nov 02 14:02:55.896343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:55.896311    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:55.898318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:55.898296    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:02:55.899503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:02:55.899486    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:03:00.167853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.167790    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll" podUID="8dc5e889-1ca4-4b79-a657-0cd1005f6b70" containerName="payload-processing" containerID="cri-o://f4319a84ac42d09d712264ed633499865d9f329de7874373aa4d4ec21ca3eaa2" gracePeriod=25
Nov 02 14:03:00.206370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.206317    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx" podUID="bfef9a88-5782-442d-a126-25b39d968683" containerName="payload-pre-processing" containerID="cri-o://fdc55804b04714347e1c3ab32ce755eac34e31598a0b53fa34c2cfe60bfa0349" gracePeriod=25
Nov 02 14:03:00.379341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.379300    5034 generic.go:356] "Generic (PLEG): container finished" podID="bfef9a88-5782-442d-a126-25b39d968683" containerID="fdc55804b04714347e1c3ab32ce755eac34e31598a0b53fa34c2cfe60bfa0349" exitCode=0
Nov 02 14:03:00.379551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.379530    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx" event={"ID":"bfef9a88-5782-442d-a126-25b39d968683","Type":"ContainerDied","Data":"fdc55804b04714347e1c3ab32ce755eac34e31598a0b53fa34c2cfe60bfa0349"}
Nov 02 14:03:00.405247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.404142    5034 generic.go:356] "Generic (PLEG): container finished" podID="8dc5e889-1ca4-4b79-a657-0cd1005f6b70" containerID="f4319a84ac42d09d712264ed633499865d9f329de7874373aa4d4ec21ca3eaa2" exitCode=0
Nov 02 14:03:00.405247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.404336    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll" event={"ID":"8dc5e889-1ca4-4b79-a657-0cd1005f6b70","Type":"ContainerDied","Data":"f4319a84ac42d09d712264ed633499865d9f329de7874373aa4d4ec21ca3eaa2"}
Nov 02 14:03:00.450524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.450480    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf" podUID="867a30da-620a-4c25-a569-c300147d5264" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:03:00.547673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.547086    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:03:00.549545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.549519    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:03:00.625065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.625029    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-kube-api-access-cjc5g\" (UniqueName: \"kubernetes.io/projected/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-kube-api-access-cjc5g\") pod \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\" (UID: \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\") "
Nov 02 14:03:00.625264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.625105    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfef9a88-5782-442d-a126-25b39d968683-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfef9a88-5782-442d-a126-25b39d968683-plugins-config-volume\") pod \"bfef9a88-5782-442d-a126-25b39d968683\" (UID: \"bfef9a88-5782-442d-a126-25b39d968683\") "
Nov 02 14:03:00.625337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.625295    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-plugins-config-volume\") pod \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\" (UID: \"8dc5e889-1ca4-4b79-a657-0cd1005f6b70\") "
Nov 02 14:03:00.625604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.625574    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfef9a88-5782-442d-a126-25b39d968683-plugins-config-volume" pod "bfef9a88-5782-442d-a126-25b39d968683" (UID: "bfef9a88-5782-442d-a126-25b39d968683"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:00.625706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.625676    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-plugins-config-volume" pod "8dc5e889-1ca4-4b79-a657-0cd1005f6b70" (UID: "8dc5e889-1ca4-4b79-a657-0cd1005f6b70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:00.625773 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.625751    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfef9a88-5782-442d-a126-25b39d968683-kube-api-access-sbnsm\" (UniqueName: \"kubernetes.io/projected/bfef9a88-5782-442d-a126-25b39d968683-kube-api-access-sbnsm\") pod \"bfef9a88-5782-442d-a126-25b39d968683\" (UID: \"bfef9a88-5782-442d-a126-25b39d968683\") "
Nov 02 14:03:00.626928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.626902    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfef9a88-5782-442d-a126-25b39d968683-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:00.626928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.626930    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:00.627763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.627740    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-kube-api-access-cjc5g" pod "8dc5e889-1ca4-4b79-a657-0cd1005f6b70" (UID: "8dc5e889-1ca4-4b79-a657-0cd1005f6b70"). InnerVolumeSpecName "kube-api-access-cjc5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:00.628184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.628165    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfef9a88-5782-442d-a126-25b39d968683-kube-api-access-sbnsm" pod "bfef9a88-5782-442d-a126-25b39d968683" (UID: "bfef9a88-5782-442d-a126-25b39d968683"). InnerVolumeSpecName "kube-api-access-sbnsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:00.728741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.728650    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbnsm\" (UniqueName: \"kubernetes.io/projected/bfef9a88-5782-442d-a126-25b39d968683-kube-api-access-sbnsm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:00.728741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:00.728685    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjc5g\" (UniqueName: \"kubernetes.io/projected/8dc5e889-1ca4-4b79-a657-0cd1005f6b70-kube-api-access-cjc5g\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:01.416852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.416826    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"
Nov 02 14:03:01.417368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.416828    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx" event={"ID":"bfef9a88-5782-442d-a126-25b39d968683","Type":"ContainerDied","Data":"1b0c331bb2bdd2c540c42bc48874eb8847efce5a3a94c4c157c1d09a1eb25163"}
Nov 02 14:03:01.417368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.416983    5034 scope.go:122] "RemoveContainer" containerID="fdc55804b04714347e1c3ab32ce755eac34e31598a0b53fa34c2cfe60bfa0349"
Nov 02 14:03:01.418952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.418927    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll" event={"ID":"8dc5e889-1ca4-4b79-a657-0cd1005f6b70","Type":"ContainerDied","Data":"80116ee0aedf5103f1b19817f5274d4752eb88d95e153605b7546cba1e020cab"}
Nov 02 14:03:01.419087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.419018    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"
Nov 02 14:03:01.429544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.429524    5034 scope.go:122] "RemoveContainer" containerID="f4319a84ac42d09d712264ed633499865d9f329de7874373aa4d4ec21ca3eaa2"
Nov 02 14:03:01.478077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.477984    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"]
Nov 02 14:03:01.489113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.489073    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9eb67b-677d64c55-wz5lx"]
Nov 02 14:03:01.518953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.518918    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"]
Nov 02 14:03:01.523809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:01.523756    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9eb67b-c8dcd568f-r2jll"]
Nov 02 14:03:03.399619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:03.399579    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc5e889-1ca4-4b79-a657-0cd1005f6b70" path="/var/lib/kubelet/pods/8dc5e889-1ca4-4b79-a657-0cd1005f6b70/volumes"
Nov 02 14:03:03.400202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:03.400183    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfef9a88-5782-442d-a126-25b39d968683" path="/var/lib/kubelet/pods/bfef9a88-5782-442d-a126-25b39d968683/volumes"
Nov 02 14:03:06.888803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:06.886390    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:03:07.228857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:07.227844    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"]
Nov 02 14:03:07.316295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:07.315896    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:07.325815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:07.317930    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d776122a-qxbw9"]
Nov 02 14:03:07.459238 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:07.455636    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be4e967-c791-491b-b9cf-7375cfd4e0bd" path="/var/lib/kubelet/pods/8be4e967-c791-491b-b9cf-7375cfd4e0bd/volumes"
Nov 02 14:03:07.920980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:07.920922    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:08.317082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:08.317027    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:08.462801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:08.459200    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:03:08.463838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:08.463757    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:08.472320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:08.472290    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:08.896651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:08.896604    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:08.898339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:08.897012    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:08.924264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:08.924182    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:03:08.926378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:08.925035    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:03:09.132539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.132206    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Nov 02 14:03:09.298479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.297878    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:09.374229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.318875    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:09.374229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.319521    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:03:09.631803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.631607    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:03:09.656805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.640602    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:03:09.656805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.648488    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:09.656805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.649245    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:09.839803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.839213    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:03:09.839803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.839286    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:03:09.975059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.961863    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:03:09.975059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.963656    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded"
Nov 02 14:03:09.975059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.963936    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:03:09.975059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.964378    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:03:09.975059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:09.964434    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:03:10.262243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.181023    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/health\": context deadline exceeded"
Nov 02 14:03:10.372827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.324047    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:03:10.602989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.602910    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:03:10.629177 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.629149    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:03:10.634182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.634146    5034 generic.go:356] "Generic (PLEG): container finished" podID="867a30da-620a-4c25-a569-c300147d5264" containerID="b37a75f8f4dc80a9104ef782db157a255c0f750a11605d26d8c24ee00a6bc6de" exitCode=0
Nov 02 14:03:10.653960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.653915    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Nov 02 14:03:10.684802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.634381    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf" event={"ID":"867a30da-620a-4c25-a569-c300147d5264","Type":"ContainerDied","Data":"b37a75f8f4dc80a9104ef782db157a255c0f750a11605d26d8c24ee00a6bc6de"}
Nov 02 14:03:10.684802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.658037    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:03:10.684802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.660304    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:10.684802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.660563    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": context deadline exceeded"
Nov 02 14:03:10.782348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.782317    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:03:10.785525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.785204    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:03:10.801575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.787300    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:03:10.801575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.800802    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:03:10.834894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.834868    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:03:10.929407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.929378    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:03:10.977802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.974839    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867a30da-620a-4c25-a569-c300147d5264-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867a30da-620a-4c25-a569-c300147d5264-plugins-config-volume\") pod \"867a30da-620a-4c25-a569-c300147d5264\" (UID: \"867a30da-620a-4c25-a569-c300147d5264\") "
Nov 02 14:03:10.977802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.975121    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867a30da-620a-4c25-a569-c300147d5264-kube-api-access-dzvsq\" (UniqueName: \"kubernetes.io/projected/867a30da-620a-4c25-a569-c300147d5264-kube-api-access-dzvsq\") pod \"867a30da-620a-4c25-a569-c300147d5264\" (UID: \"867a30da-620a-4c25-a569-c300147d5264\") "
Nov 02 14:03:10.977802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.977159    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867a30da-620a-4c25-a569-c300147d5264-plugins-config-volume" pod "867a30da-620a-4c25-a569-c300147d5264" (UID: "867a30da-620a-4c25-a569-c300147d5264"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:10.986315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:10.986286    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867a30da-620a-4c25-a569-c300147d5264-kube-api-access-dzvsq" pod "867a30da-620a-4c25-a569-c300147d5264" (UID: "867a30da-620a-4c25-a569-c300147d5264"). InnerVolumeSpecName "kube-api-access-dzvsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:11.115879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:11.114721    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzvsq\" (UniqueName: \"kubernetes.io/projected/867a30da-620a-4c25-a569-c300147d5264-kube-api-access-dzvsq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.115879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:11.114758    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/867a30da-620a-4c25-a569-c300147d5264-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:11.454582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:11.453814    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf" podUID="867a30da-620a-4c25-a569-c300147d5264" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.128:50052\" within 1s: context deadline exceeded"
Nov 02 14:03:11.454582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:11.453913    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:03:11.454582 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:03:11.454079    5034 logging.go:55] [core] [Channel #1266 SubChannel #1267]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.128:50052", ServerName: "10.132.0.128:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.128:50052: operation was canceled"
Nov 02 14:03:11.668453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:11.668424    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"
Nov 02 14:03:11.668673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:11.668439    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf" event={"ID":"867a30da-620a-4c25-a569-c300147d5264","Type":"ContainerDied","Data":"8be3832243f250439b094c71d2be8e8e59a98aef2a96b925e185b88792f36fdf"}
Nov 02 14:03:11.668673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:11.668540    5034 scope.go:122] "RemoveContainer" containerID="b37a75f8f4dc80a9104ef782db157a255c0f750a11605d26d8c24ee00a6bc6de"
Nov 02 14:03:11.751190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:11.751158    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"]
Nov 02 14:03:11.756308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:11.756282    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d776122a-544ff95bf5-gpxxf"]
Nov 02 14:03:13.401958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:13.401915    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867a30da-620a-4c25-a569-c300147d5264" path="/var/lib/kubelet/pods/867a30da-620a-4c25-a569-c300147d5264/volumes"
Nov 02 14:03:16.187960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.187915    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"]
Nov 02 14:03:16.189636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.189567    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy" containerID="cri-o://ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa" gracePeriod=30
Nov 02 14:03:16.541982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.523512    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"]
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543309    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8dc5e889-1ca4-4b79-a657-0cd1005f6b70" containerName="payload-processing"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543339    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc5e889-1ca4-4b79-a657-0cd1005f6b70" containerName="payload-processing"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543399    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bfef9a88-5782-442d-a126-25b39d968683" containerName="payload-pre-processing"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543408    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfef9a88-5782-442d-a126-25b39d968683" containerName="payload-pre-processing"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543450    5034 cpu_manager.go:414] "containerMap: removing container" podUID="867a30da-620a-4c25-a569-c300147d5264" containerName="payload-pre-processing"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543457    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867a30da-620a-4c25-a569-c300147d5264" containerName="payload-pre-processing"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543470    5034 cpu_manager.go:414] "containerMap: removing container" podUID="06467082-aef9-4454-9f96-c8f21c174b9b" containerName="revoke-keys"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543477    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06467082-aef9-4454-9f96-c8f21c174b9b" containerName="revoke-keys"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543649    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfef9a88-5782-442d-a126-25b39d968683" containerName="payload-pre-processing"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543669    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="867a30da-620a-4c25-a569-c300147d5264" containerName="payload-pre-processing"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543680    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="06467082-aef9-4454-9f96-c8f21c174b9b" containerName="revoke-keys"
Nov 02 14:03:16.587878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.543693    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc5e889-1ca4-4b79-a657-0cd1005f6b70" containerName="payload-processing"
Nov 02 14:03:16.621765 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.621738    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"]
Nov 02 14:03:16.621946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.621876    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:16.640133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.640083    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d776122a-dockercfg-7tptq\""
Nov 02 14:03:16.643814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.643757    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d776122a\""
Nov 02 14:03:16.803934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.803844    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f386e34-64f2-4f82-876a-b40c0287ad32-plugins-config-volume\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2\" (UID: \"5f386e34-64f2-4f82-876a-b40c0287ad32\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:16.803934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.803920    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2mh\" (UniqueName: \"kubernetes.io/projected/5f386e34-64f2-4f82-876a-b40c0287ad32-kube-api-access-rp2mh\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2\" (UID: \"5f386e34-64f2-4f82-876a-b40c0287ad32\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:16.910802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.907304    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f386e34-64f2-4f82-876a-b40c0287ad32-plugins-config-volume\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2\" (UID: \"5f386e34-64f2-4f82-876a-b40c0287ad32\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:16.910802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.907386    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2mh\" (UniqueName: \"kubernetes.io/projected/5f386e34-64f2-4f82-876a-b40c0287ad32-kube-api-access-rp2mh\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2\" (UID: \"5f386e34-64f2-4f82-876a-b40c0287ad32\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:16.910802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.908768    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f386e34-64f2-4f82-876a-b40c0287ad32-plugins-config-volume\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2\" (UID: \"5f386e34-64f2-4f82-876a-b40c0287ad32\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:16.939078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.939042    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2mh\" (UniqueName: \"kubernetes.io/projected/5f386e34-64f2-4f82-876a-b40c0287ad32-kube-api-access-rp2mh\") pod \"payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2\" (UID: \"5f386e34-64f2-4f82-876a-b40c0287ad32\") " pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:16.951526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:16.951487    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:17.330596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:17.330568    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"]
Nov 02 14:03:17.338261 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:03:17.338218    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f386e34_64f2_4f82_876a_b40c0287ad32.slice/crio-702621e950dc45ad519ba8e9cd22d2da9ca19a78f7dc67994bb6f2b33631bc2d WatchSource:0}: Error finding container 702621e950dc45ad519ba8e9cd22d2da9ca19a78f7dc67994bb6f2b33631bc2d: Status 404 returned error can't find the container with id 702621e950dc45ad519ba8e9cd22d2da9ca19a78f7dc67994bb6f2b33631bc2d
Nov 02 14:03:17.814724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:17.814673    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2" event={"ID":"5f386e34-64f2-4f82-876a-b40c0287ad32","Type":"ContainerStarted","Data":"13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b"}
Nov 02 14:03:17.814972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:17.814811    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2" event={"ID":"5f386e34-64f2-4f82-876a-b40c0287ad32","Type":"ContainerStarted","Data":"702621e950dc45ad519ba8e9cd22d2da9ca19a78f7dc67994bb6f2b33631bc2d"}
Nov 02 14:03:17.814972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:17.814856    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:17.874253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:17.874192    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2" podStartSLOduration=1.874175656 podStartE2EDuration="1.874175656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:17.868497073 +0000 UTC m=+1188.619801508" watchObservedRunningTime="2026-11-02 14:03:17.874175656 +0000 UTC m=+1188.625480096"
Nov 02 14:03:18.417896 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:03:18.417854    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:03:18.829603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:18.829571    5034 generic.go:356] "Generic (PLEG): container finished" podID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerID="e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597" exitCode=0
Nov 02 14:03:18.829833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:18.829660    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" event={"ID":"dd6328bd-5689-4d4e-9fa0-3ab417072485","Type":"ContainerDied","Data":"e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597"}
Nov 02 14:03:18.830358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:18.830337    5034 scope.go:122] "RemoveContainer" containerID="e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597"
Nov 02 14:03:18.832036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:18.832006    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:03:18.832135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:18.832087    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:03:18.832135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:18.832129    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:03:18.965914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:18.965887    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:03:19.840961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:19.840926    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" event={"ID":"dd6328bd-5689-4d4e-9fa0-3ab417072485","Type":"ContainerStarted","Data":"f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04"}
Nov 02 14:03:20.027947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.027916    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-f2hmh-rrqpl"]
Nov 02 14:03:20.032507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.032488    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.052189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.052151    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-n6hr6\""
Nov 02 14:03:20.056116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.056091    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f2hmh-rrqpl"]
Nov 02 14:03:20.144580 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.144495    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:03:20.145255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.145224    5034 patch_prober.go:29] interesting pod/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:03:20.145515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.145488    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 14:03:20.211319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.211274    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-archives-path\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.211647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.211625    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-service-ca-bundle\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.211708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.211663    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-trusted-ca-bundle\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.211787 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.211757    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsg4m\" (UniqueName: \"kubernetes.io/projected/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-kube-api-access-jsg4m\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.312850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.312812    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-service-ca-bundle\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.312850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.312860    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-trusted-ca-bundle\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.313135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.312889    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsg4m\" (UniqueName: \"kubernetes.io/projected/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-kube-api-access-jsg4m\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.313135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.312979    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-archives-path\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.313497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.313471    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-archives-path\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.313689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.313666    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-service-ca-bundle\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.313832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.313808    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-trusted-ca-bundle\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.339024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.339002    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsg4m\" (UniqueName: \"kubernetes.io/projected/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-kube-api-access-jsg4m\") pod \"periodic-gathering-f2hmh-rrqpl\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") " pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.345945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.345921    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:03:20.755865 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:03:20.755825    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadb576c_9eeb_4d28_8afb_5ac016a0bbc9.slice/crio-b3eab483400d6ce0c381ec011e98dffefa2105e9629a35a67fdc403d1e5482d4 WatchSource:0}: Error finding container b3eab483400d6ce0c381ec011e98dffefa2105e9629a35a67fdc403d1e5482d4: Status 404 returned error can't find the container with id b3eab483400d6ce0c381ec011e98dffefa2105e9629a35a67fdc403d1e5482d4
Nov 02 14:03:20.757771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.757748    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-f2hmh-rrqpl"]
Nov 02 14:03:20.851116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:20.851081    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f2hmh-rrqpl" event={"ID":"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9","Type":"ContainerStarted","Data":"b3eab483400d6ce0c381ec011e98dffefa2105e9629a35a67fdc403d1e5482d4"}
Nov 02 14:03:21.145730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.145142    5034 patch_prober.go:29] interesting pod/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:03:21.145730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.145191    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 14:03:21.648690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.648658    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:03:21.781622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.770359    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-podinfo\") pod \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") "
Nov 02 14:03:21.781622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.770424    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-data\") pod \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") "
Nov 02 14:03:21.781622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.770470    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-socket\") pod \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") "
Nov 02 14:03:21.781622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.770504    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-kube-api-access-4wbhw\") pod \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") "
Nov 02 14:03:21.781622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.770573    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-credential-socket\") pod \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") "
Nov 02 14:03:21.781622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.770744    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-certs\") pod \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") "
Nov 02 14:03:21.781622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.774914    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-envoy\") pod \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") "
Nov 02 14:03:21.781622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.775002    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-token\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-token\") pod \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") "
Nov 02 14:03:21.781622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.775045    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istiod-ca-cert\") pod \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\" (UID: \"f34e96f8-c765-4ad6-8cd4-aec1f564ca78\") "
Nov 02 14:03:21.804838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.796881    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-certs" pod "f34e96f8-c765-4ad6-8cd4-aec1f564ca78" (UID: "f34e96f8-c765-4ad6-8cd4-aec1f564ca78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:21.828932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.813000    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istiod-ca-cert" pod "f34e96f8-c765-4ad6-8cd4-aec1f564ca78" (UID: "f34e96f8-c765-4ad6-8cd4-aec1f564ca78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:21.828932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.822063    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-credential-socket" pod "f34e96f8-c765-4ad6-8cd4-aec1f564ca78" (UID: "f34e96f8-c765-4ad6-8cd4-aec1f564ca78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:21.828932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.828862    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-data" pod "f34e96f8-c765-4ad6-8cd4-aec1f564ca78" (UID: "f34e96f8-c765-4ad6-8cd4-aec1f564ca78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:21.914141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.914026    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:21.914141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.914057    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:21.914141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.914075    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:21.914141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.914091    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:22.021891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.984072    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-socket" pod "f34e96f8-c765-4ad6-8cd4-aec1f564ca78" (UID: "f34e96f8-c765-4ad6-8cd4-aec1f564ca78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:22.021891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:21.988243    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-podinfo" pod "f34e96f8-c765-4ad6-8cd4-aec1f564ca78" (UID: "f34e96f8-c765-4ad6-8cd4-aec1f564ca78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:03:22.021891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.002122    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-kube-api-access-4wbhw" pod "f34e96f8-c765-4ad6-8cd4-aec1f564ca78" (UID: "f34e96f8-c765-4ad6-8cd4-aec1f564ca78"). InnerVolumeSpecName "kube-api-access-4wbhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:22.021891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.002215    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-token" pod "f34e96f8-c765-4ad6-8cd4-aec1f564ca78" (UID: "f34e96f8-c765-4ad6-8cd4-aec1f564ca78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:22.021891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.002268    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-envoy" pod "f34e96f8-c765-4ad6-8cd4-aec1f564ca78" (UID: "f34e96f8-c765-4ad6-8cd4-aec1f564ca78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:22.029882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.029374    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbhw\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-kube-api-access-4wbhw\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:22.029882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.029400    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:22.029882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.029410    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:22.029882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.029417    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:22.029882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.029426    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34e96f8-c765-4ad6-8cd4-aec1f564ca78-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:22.037456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.037389    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f2hmh-rrqpl" event={"ID":"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9","Type":"ContainerStarted","Data":"99ef4b9c07185b93095bb55d7c243e9e305fc0d12a0a49045e7943fa035f29f7"}
Nov 02 14:03:22.041730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.041119    5034 generic.go:356] "Generic (PLEG): container finished" podID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerID="ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa" exitCode=0
Nov 02 14:03:22.041730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.041216    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"
Nov 02 14:03:22.041730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.041253    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" event={"ID":"f34e96f8-c765-4ad6-8cd4-aec1f564ca78","Type":"ContainerDied","Data":"ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa"}
Nov 02 14:03:22.041730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.041297    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664" event={"ID":"f34e96f8-c765-4ad6-8cd4-aec1f564ca78","Type":"ContainerDied","Data":"ad2d108ebad4169b99cd0f79617bcfb72235b34a5e3fcf6647d668e4315c50e5"}
Nov 02 14:03:22.041730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.041326    5034 scope.go:122] "RemoveContainer" containerID="ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa"
Nov 02 14:03:22.056240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.056221    5034 scope.go:122] "RemoveContainer" containerID="ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9"
Nov 02 14:03:22.066785 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.066749    5034 scope.go:122] "RemoveContainer" containerID="ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa"
Nov 02 14:03:22.067080 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:03:22.067056    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa\": container with ID starting with ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa not found: ID does not exist" containerID="ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa"
Nov 02 14:03:22.067149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.067089    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa"} err="failed to get container status \"ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa\": rpc error: code = NotFound desc = could not find container \"ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa\": container with ID starting with ee33e33ae574a286604e3cbed8f65db376d24ff9928aba0ddfe50962d7d956fa not found: ID does not exist"
Nov 02 14:03:22.067149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.067105    5034 scope.go:122] "RemoveContainer" containerID="ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9"
Nov 02 14:03:22.067351 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:03:22.067332    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9\": container with ID starting with ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9 not found: ID does not exist" containerID="ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9"
Nov 02 14:03:22.067408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.067357    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9"} err="failed to get container status \"ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9\": rpc error: code = NotFound desc = could not find container \"ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9\": container with ID starting with ffce4afc47c661f8005fc326d3ed66672cf38956774aa16f6b91f8e000d3f5b9 not found: ID does not exist"
Nov 02 14:03:22.110652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.110621    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"]
Nov 02 14:03:22.115561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.115535    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa44ce7-5wk9d"]
Nov 02 14:03:22.119616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.119573    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-f2hmh-rrqpl" podStartSLOduration=3.119560897 podStartE2EDuration="3.119560897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:22.104130893 +0000 UTC m=+1192.855435332" watchObservedRunningTime="2026-11-02 14:03:22.119560897 +0000 UTC m=+1192.870865334"
Nov 02 14:03:22.145225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.145193    5034 patch_prober.go:29] interesting pod/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:03:22.145317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.145254    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 14:03:22.147898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.147878    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"]
Nov 02 14:03:22.160950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:22.160920    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4c6741-openshift-default-7b5c9b9775-hr664"]
Nov 02 14:03:23.144525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:23.144490    5034 patch_prober.go:29] interesting pod/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:03:23.144969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:23.144542    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 14:03:23.410076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:23.409983    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec00317-2f7d-49b7-b976-1bdd135c6352" path="/var/lib/kubelet/pods/9ec00317-2f7d-49b7-b976-1bdd135c6352/volumes"
Nov 02 14:03:23.410817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:23.410793    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" path="/var/lib/kubelet/pods/f34e96f8-c765-4ad6-8cd4-aec1f564ca78/volumes"
Nov 02 14:03:24.145274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.145239    5034 patch_prober.go:29] interesting pod/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:03:24.145660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.145291    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 14:03:24.602881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.602836    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"]
Nov 02 14:03:24.603666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.603647    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy"
Nov 02 14:03:24.603795 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.603668    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy"
Nov 02 14:03:24.603878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.603865    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy"
Nov 02 14:03:24.603938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.603883    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy"
Nov 02 14:03:24.609947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.609924    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"]
Nov 02 14:03:24.610095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.610013    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:24.631948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.631919    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8d5eec-dockercfg-4g2qs\""
Nov 02 14:03:24.631948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.631919    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8d5eec\""
Nov 02 14:03:24.647245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.647220    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"]
Nov 02 14:03:24.648446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.648425    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy"
Nov 02 14:03:24.648502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.648450    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34e96f8-c765-4ad6-8cd4-aec1f564ca78" containerName="istio-proxy"
Nov 02 14:03:24.653175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.653154    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:24.677207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.669983    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62cg\" (UniqueName: \"kubernetes.io/projected/38608c61-0ac5-41fe-b2eb-04f7f83c387f-kube-api-access-k62cg\") pod \"payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c\" (UID: \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:24.677207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.670064    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38608c61-0ac5-41fe-b2eb-04f7f83c387f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c\" (UID: \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:24.830800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.828726    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38608c61-0ac5-41fe-b2eb-04f7f83c387f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c\" (UID: \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:24.830800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.829950    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38608c61-0ac5-41fe-b2eb-04f7f83c387f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c\" (UID: \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:24.850806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.839919    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62cg\" (UniqueName: \"kubernetes.io/projected/38608c61-0ac5-41fe-b2eb-04f7f83c387f-kube-api-access-k62cg\") pod \"payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c\" (UID: \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:24.943688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:24.943590    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62cg\" (UniqueName: \"kubernetes.io/projected/38608c61-0ac5-41fe-b2eb-04f7f83c387f-kube-api-access-k62cg\") pod \"payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c\" (UID: \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\") " pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:25.015139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.001191    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadceb97-3b8e-49a1-99fc-0324f0c50c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx\" (UID: \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:25.036016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.035974    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wjtc\" (UniqueName: \"kubernetes.io/projected/fadceb97-3b8e-49a1-99fc-0324f0c50c63-kube-api-access-6wjtc\") pod \"payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx\" (UID: \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:25.049584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.041507    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"]
Nov 02 14:03:25.081500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.081466    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"]
Nov 02 14:03:25.113108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.113078    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"]
Nov 02 14:03:25.137856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.137810    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wjtc\" (UniqueName: \"kubernetes.io/projected/fadceb97-3b8e-49a1-99fc-0324f0c50c63-kube-api-access-6wjtc\") pod \"payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx\" (UID: \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:25.138220 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.138201    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadceb97-3b8e-49a1-99fc-0324f0c50c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx\" (UID: \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:25.138910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.138889    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadceb97-3b8e-49a1-99fc-0324f0c50c63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx\" (UID: \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:25.144607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.144573    5034 patch_prober.go:29] interesting pod/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Nov 02 14:03:25.144718 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.144641    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Nov 02 14:03:25.164830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.164796    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wjtc\" (UniqueName: \"kubernetes.io/projected/fadceb97-3b8e-49a1-99fc-0324f0c50c63-kube-api-access-6wjtc\") pod \"payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx\" (UID: \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:25.230617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.230528    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:25.429832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.429770    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:25.590518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.590487    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"]
Nov 02 14:03:25.592310 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:03:25.592284    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38608c61_0ac5_41fe_b2eb_04f7f83c387f.slice/crio-230e406737ed921b5f38851e3644cefa51fbff2539c7f71842c38c1af89af213 WatchSource:0}: Error finding container 230e406737ed921b5f38851e3644cefa51fbff2539c7f71842c38c1af89af213: Status 404 returned error can't find the container with id 230e406737ed921b5f38851e3644cefa51fbff2539c7f71842c38c1af89af213
Nov 02 14:03:25.813807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:25.813351    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"]
Nov 02 14:03:26.084809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:26.083185    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx" event={"ID":"fadceb97-3b8e-49a1-99fc-0324f0c50c63","Type":"ContainerStarted","Data":"269259489f773817844db96946a8832d7a765a4af9c543c3b7519878b2876d07"}
Nov 02 14:03:26.089577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:26.087301    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c" event={"ID":"38608c61-0ac5-41fe-b2eb-04f7f83c387f","Type":"ContainerStarted","Data":"cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987"}
Nov 02 14:03:26.089577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:26.087582    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:26.089577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:26.087615    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c" event={"ID":"38608c61-0ac5-41fe-b2eb-04f7f83c387f","Type":"ContainerStarted","Data":"230e406737ed921b5f38851e3644cefa51fbff2539c7f71842c38c1af89af213"}
Nov 02 14:03:26.130656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:26.130609    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c" podStartSLOduration=2.130597494 podStartE2EDuration="2.130597494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:26.129867859 +0000 UTC m=+1196.881172298" watchObservedRunningTime="2026-11-02 14:03:26.130597494 +0000 UTC m=+1196.881901931"
Nov 02 14:03:26.148962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:26.148935    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:03:26.149410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:26.149398    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:03:26.150120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:26.150101    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:03:27.099051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:27.099007    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx" event={"ID":"fadceb97-3b8e-49a1-99fc-0324f0c50c63","Type":"ContainerStarted","Data":"a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10"}
Nov 02 14:03:27.146403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:27.146359    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx" podStartSLOduration=3.146345416 podStartE2EDuration="3.146345416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:27.143025426 +0000 UTC m=+1197.894329866" watchObservedRunningTime="2026-11-02 14:03:27.146345416 +0000 UTC m=+1197.897649853"
Nov 02 14:03:28.109460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:28.109416    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:28.833427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:28.833400    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:03:29.328320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:29.328287    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"]
Nov 02 14:03:29.577311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:29.577275    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:03:29.577608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:29.577274    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:03:30.103326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.084837    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd" podUID="2f16e8b8-d839-40fc-95ca-544dee0fb8b5" containerName="payload-processing" containerID="cri-o://afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b" gracePeriod=25
Nov 02 14:03:30.123828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.120531    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn" podUID="6c508bca-6066-4326-bf21-54460253ad4d" containerName="payload-pre-processing" containerID="cri-o://fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6" gracePeriod=25
Nov 02 14:03:30.514294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.514266    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:03:30.538689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.538667    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:03:30.614631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.614596    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-kube-api-access-w6wdn\" (UniqueName: \"kubernetes.io/projected/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-kube-api-access-w6wdn\") pod \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\" (UID: \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\") "
Nov 02 14:03:30.614864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.614680    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c508bca-6066-4326-bf21-54460253ad4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c508bca-6066-4326-bf21-54460253ad4d-plugins-config-volume\") pod \"6c508bca-6066-4326-bf21-54460253ad4d\" (UID: \"6c508bca-6066-4326-bf21-54460253ad4d\") "
Nov 02 14:03:30.614864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.614710    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-plugins-config-volume\") pod \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\" (UID: \"2f16e8b8-d839-40fc-95ca-544dee0fb8b5\") "
Nov 02 14:03:30.615016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.615005    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c508bca-6066-4326-bf21-54460253ad4d-kube-api-access-lkxxr\" (UniqueName: \"kubernetes.io/projected/6c508bca-6066-4326-bf21-54460253ad4d-kube-api-access-lkxxr\") pod \"6c508bca-6066-4326-bf21-54460253ad4d\" (UID: \"6c508bca-6066-4326-bf21-54460253ad4d\") "
Nov 02 14:03:30.616877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.616586    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c508bca-6066-4326-bf21-54460253ad4d-plugins-config-volume" pod "6c508bca-6066-4326-bf21-54460253ad4d" (UID: "6c508bca-6066-4326-bf21-54460253ad4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:30.617470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.617421    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-plugins-config-volume" pod "2f16e8b8-d839-40fc-95ca-544dee0fb8b5" (UID: "2f16e8b8-d839-40fc-95ca-544dee0fb8b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:30.618652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.618616    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c508bca-6066-4326-bf21-54460253ad4d-kube-api-access-lkxxr" pod "6c508bca-6066-4326-bf21-54460253ad4d" (UID: "6c508bca-6066-4326-bf21-54460253ad4d"). InnerVolumeSpecName "kube-api-access-lkxxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:30.620157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.620124    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-kube-api-access-w6wdn" pod "2f16e8b8-d839-40fc-95ca-544dee0fb8b5" (UID: "2f16e8b8-d839-40fc-95ca-544dee0fb8b5"). InnerVolumeSpecName "kube-api-access-w6wdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:30.717699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.717606    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6wdn\" (UniqueName: \"kubernetes.io/projected/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-kube-api-access-w6wdn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:30.717699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.717648    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c508bca-6066-4326-bf21-54460253ad4d-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:30.717699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.717658    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f16e8b8-d839-40fc-95ca-544dee0fb8b5-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:30.717699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:30.717667    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkxxr\" (UniqueName: \"kubernetes.io/projected/6c508bca-6066-4326-bf21-54460253ad4d-kube-api-access-lkxxr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:31.149770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.149733    5034 generic.go:356] "Generic (PLEG): container finished" podID="2f16e8b8-d839-40fc-95ca-544dee0fb8b5" containerID="afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b" exitCode=0
Nov 02 14:03:31.150013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.149804    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd" event={"ID":"2f16e8b8-d839-40fc-95ca-544dee0fb8b5","Type":"ContainerDied","Data":"afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b"}
Nov 02 14:03:31.150013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.149812    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"
Nov 02 14:03:31.150013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.149876    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd" event={"ID":"2f16e8b8-d839-40fc-95ca-544dee0fb8b5","Type":"ContainerDied","Data":"5848200737e045a4c7a5a33f4e81ee3467bf9baa96064d653f95cfe341998f34"}
Nov 02 14:03:31.150013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.149926    5034 scope.go:122] "RemoveContainer" containerID="afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b"
Nov 02 14:03:31.152210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.152189    5034 generic.go:356] "Generic (PLEG): container finished" podID="6c508bca-6066-4326-bf21-54460253ad4d" containerID="fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6" exitCode=0
Nov 02 14:03:31.152359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.152342    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn" event={"ID":"6c508bca-6066-4326-bf21-54460253ad4d","Type":"ContainerDied","Data":"fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6"}
Nov 02 14:03:31.152431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.152385    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn" event={"ID":"6c508bca-6066-4326-bf21-54460253ad4d","Type":"ContainerDied","Data":"f4ccd22506f953417b4d5b77dcb4e799e8785032f800086cc5a90808d2e5660d"}
Nov 02 14:03:31.152502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.152454    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"
Nov 02 14:03:31.164523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.164507    5034 scope.go:122] "RemoveContainer" containerID="afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b"
Nov 02 14:03:31.164790 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:03:31.164762    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b\": container with ID starting with afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b not found: ID does not exist" containerID="afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b"
Nov 02 14:03:31.164846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.164815    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b"} err="failed to get container status \"afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b\": rpc error: code = NotFound desc = could not find container \"afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b\": container with ID starting with afdd7f7f6f9ee65bc77508466b9c7b4036a6a50dcd5102aad227a0467b81ce7b not found: ID does not exist"
Nov 02 14:03:31.164846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.164832    5034 scope.go:122] "RemoveContainer" containerID="fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6"
Nov 02 14:03:31.179857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.179834    5034 scope.go:122] "RemoveContainer" containerID="fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6"
Nov 02 14:03:31.180199 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:03:31.180174    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6\": container with ID starting with fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6 not found: ID does not exist" containerID="fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6"
Nov 02 14:03:31.180284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.180209    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6"} err="failed to get container status \"fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6\": rpc error: code = NotFound desc = could not find container \"fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6\": container with ID starting with fa9b0d3004d18c95c996a00451a1e8c1745876e51ace11eb07cd5e893c068fc6 not found: ID does not exist"
Nov 02 14:03:31.208935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.208912    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"]
Nov 02 14:03:31.228077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.228039    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-981903-6db686fbb5-lvmdd"]
Nov 02 14:03:31.262248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.262222    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"]
Nov 02 14:03:31.266438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.266415    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-981903-9d96b7b7-b98bn"]
Nov 02 14:03:31.398385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.398351    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f16e8b8-d839-40fc-95ca-544dee0fb8b5" path="/var/lib/kubelet/pods/2f16e8b8-d839-40fc-95ca-544dee0fb8b5/volumes"
Nov 02 14:03:31.398863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:31.398835    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c508bca-6066-4326-bf21-54460253ad4d" path="/var/lib/kubelet/pods/6c508bca-6066-4326-bf21-54460253ad4d/volumes"
Nov 02 14:03:34.329085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.329040    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z" podUID="4c7e0a7f-60c5-41ee-9c67-8b019de192f9" containerName="payload-processing" containerID="cri-o://ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218" gracePeriod=25
Nov 02 14:03:34.542805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.539295    5034 scope.go:122] "RemoveContainer" containerID="8768532411feb40c2bccff9f7d63b79d3fedff527c8cdcc25840eebdfedbfeaa"
Nov 02 14:03:34.566679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.566654    5034 scope.go:122] "RemoveContainer" containerID="941a3e7a9aeceb8fdae9f2ed8b863746e90d8cdef100d8c921424ca59db17970"
Nov 02 14:03:34.594479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.594200    5034 scope.go:122] "RemoveContainer" containerID="88301783dac8364bbd3b6e03b23d1254e59aadac2198000261db0a62f42f3943"
Nov 02 14:03:34.608987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.608747    5034 scope.go:122] "RemoveContainer" containerID="6e4e213f78417c36a7b1defb2aca4de23a9283d089402b22ff2d4ff900b5cad4"
Nov 02 14:03:34.634439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.634014    5034 scope.go:122] "RemoveContainer" containerID="0599dde52c78d81f0c28b756899ad0868362f53c5445f818b22c5dedba559052"
Nov 02 14:03:34.819762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.819731    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:03:34.909151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.909063    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-plugins-config-volume\") pod \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\" (UID: \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\") "
Nov 02 14:03:34.909297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.909153    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-kube-api-access-6v9pl\" (UniqueName: \"kubernetes.io/projected/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-kube-api-access-6v9pl\") pod \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\" (UID: \"4c7e0a7f-60c5-41ee-9c67-8b019de192f9\") "
Nov 02 14:03:34.909497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.909473    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-plugins-config-volume" pod "4c7e0a7f-60c5-41ee-9c67-8b019de192f9" (UID: "4c7e0a7f-60c5-41ee-9c67-8b019de192f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:34.909889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.909867    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:34.911442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:34.911414    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-kube-api-access-6v9pl" pod "4c7e0a7f-60c5-41ee-9c67-8b019de192f9" (UID: "4c7e0a7f-60c5-41ee-9c67-8b019de192f9"). InnerVolumeSpecName "kube-api-access-6v9pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:35.012112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.012048    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9pl\" (UniqueName: \"kubernetes.io/projected/4c7e0a7f-60c5-41ee-9c67-8b019de192f9-kube-api-access-6v9pl\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:35.196929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.196844    5034 generic.go:356] "Generic (PLEG): container finished" podID="4c7e0a7f-60c5-41ee-9c67-8b019de192f9" containerID="ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218" exitCode=0
Nov 02 14:03:35.196929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.196899    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"
Nov 02 14:03:35.197203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.196974    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z" event={"ID":"4c7e0a7f-60c5-41ee-9c67-8b019de192f9","Type":"ContainerDied","Data":"ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218"}
Nov 02 14:03:35.197203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.197038    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z" event={"ID":"4c7e0a7f-60c5-41ee-9c67-8b019de192f9","Type":"ContainerDied","Data":"4e63888a9bdbbeac81ede15d83748aaa30986c93a0139757c82596fd2e4a469c"}
Nov 02 14:03:35.197203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.197076    5034 scope.go:122] "RemoveContainer" containerID="ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218"
Nov 02 14:03:35.229184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.226682    5034 scope.go:122] "RemoveContainer" containerID="ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218"
Nov 02 14:03:35.229184 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:03:35.227193    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218\": container with ID starting with ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218 not found: ID does not exist" containerID="ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218"
Nov 02 14:03:35.229184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.227231    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218"} err="failed to get container status \"ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218\": rpc error: code = NotFound desc = could not find container \"ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218\": container with ID starting with ecfe8bb8a8376832fe95256fe6153246554e2d3e10ef8a8eb89bdbeb95081218 not found: ID does not exist"
Nov 02 14:03:35.307636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.307606    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"]
Nov 02 14:03:35.319817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.319788    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-5e9a9d-5c67b5567-zkc9z"]
Nov 02 14:03:35.397235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:35.397203    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c7e0a7f-60c5-41ee-9c67-8b019de192f9" path="/var/lib/kubelet/pods/4c7e0a7f-60c5-41ee-9c67-8b019de192f9/volumes"
Nov 02 14:03:37.102440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:37.102413    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:03:39.125430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:39.125398    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:03:42.494955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.494877    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"]
Nov 02 14:03:42.495755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.495734    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4c7e0a7f-60c5-41ee-9c67-8b019de192f9" containerName="payload-processing"
Nov 02 14:03:42.495755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.495755    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c7e0a7f-60c5-41ee-9c67-8b019de192f9" containerName="payload-processing"
Nov 02 14:03:42.495921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.495771    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2f16e8b8-d839-40fc-95ca-544dee0fb8b5" containerName="payload-processing"
Nov 02 14:03:42.495921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.495791    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f16e8b8-d839-40fc-95ca-544dee0fb8b5" containerName="payload-processing"
Nov 02 14:03:42.495921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.495801    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6c508bca-6066-4326-bf21-54460253ad4d" containerName="payload-pre-processing"
Nov 02 14:03:42.495921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.495806    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c508bca-6066-4326-bf21-54460253ad4d" containerName="payload-pre-processing"
Nov 02 14:03:42.496053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.495925    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c7e0a7f-60c5-41ee-9c67-8b019de192f9" containerName="payload-processing"
Nov 02 14:03:42.496053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.495934    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f16e8b8-d839-40fc-95ca-544dee0fb8b5" containerName="payload-processing"
Nov 02 14:03:42.496053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.495943    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c508bca-6066-4326-bf21-54460253ad4d" containerName="payload-pre-processing"
Nov 02 14:03:42.501239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.501219    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.534364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.534336    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 14:03:42.534364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.534350    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 14:03:42.534364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.534342    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5wglw\""
Nov 02 14:03:42.534364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.534346    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 14:03:42.539624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.539604    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"]
Nov 02 14:03:42.565157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.565132    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.565157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.565167    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvxxn\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-kube-api-access-gvxxn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.565490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.565472    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.565615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.565598    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.565691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.565670    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.565729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.565722    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.565796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.565757    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.666912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.666880    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.667137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.666983    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.667137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.667038    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.667137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.667082    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.667137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.667126    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.667400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.667173    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.667400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.667202    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvxxn\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-kube-api-access-gvxxn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.667933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.667905    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.669909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.669884    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.670069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.669958    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.670158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.670089    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.670298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.670276    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.691791 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.691758    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.691954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.691931    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvxxn\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-kube-api-access-gvxxn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qxkc5\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:42.815489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:42.815426    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:43.149987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:43.149906    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"]
Nov 02 14:03:43.158342 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:03:43.158315    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd818ae64_0d7e_4a50_8200_b77e0dcdf8e7.slice/crio-72e9eecff9744dbd92cd46da80d21ae953292a37a240db91394cd04ce38ce446 WatchSource:0}: Error finding container 72e9eecff9744dbd92cd46da80d21ae953292a37a240db91394cd04ce38ce446: Status 404 returned error can't find the container with id 72e9eecff9744dbd92cd46da80d21ae953292a37a240db91394cd04ce38ce446
Nov 02 14:03:43.380088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:43.380051    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5" event={"ID":"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7","Type":"ContainerStarted","Data":"72e9eecff9744dbd92cd46da80d21ae953292a37a240db91394cd04ce38ce446"}
Nov 02 14:03:47.333503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:47.333437    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:03:47.333998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:47.333518    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:03:48.434866 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:03:48.429855    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:03:48.448882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:48.448796    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5" event={"ID":"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7","Type":"ContainerStarted","Data":"8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91"}
Nov 02 14:03:48.449424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:48.449405    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:48.452678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:48.452639    5034 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qxkc5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:03:48.455003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:48.454491    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5" podUID="d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:03:48.570898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:48.570831    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5" podStartSLOduration=2.39904303 podStartE2EDuration="6.570812802s" totalImagesPullingTime="4.171769772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:03:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:03:48.570338484 +0000 UTC m=+1219.321642923" watchObservedRunningTime="2026-11-02 14:03:48.570812802 +0000 UTC m=+1219.322117245"
Nov 02 14:03:49.466745 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:49.466714    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:03:50.560290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:50.560256    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"]
Nov 02 14:03:50.562965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:50.562917    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy" containerID="cri-o://eca2a0d221d47b45122059b4b51ef7c9fc3861755c5ce5c9a68fc15750449e39" gracePeriod=30
Nov 02 14:03:55.699065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:55.699020    5034 generic.go:356] "Generic (PLEG): container finished" podID="2dc72acb-c794-4eb1-aff0-f23697179813" containerID="eca2a0d221d47b45122059b4b51ef7c9fc3861755c5ce5c9a68fc15750449e39" exitCode=0
Nov 02 14:03:55.699749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:55.699719    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" event={"ID":"2dc72acb-c794-4eb1-aff0-f23697179813","Type":"ContainerDied","Data":"eca2a0d221d47b45122059b4b51ef7c9fc3861755c5ce5c9a68fc15750449e39"}
Nov 02 14:03:55.699947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:55.699929    5034 scope.go:122] "RemoveContainer" containerID="5054f5ee21ef0471e0cfbe04a434b98f2ef96dce9d9b94114ae7710aac36ebe5"
Nov 02 14:03:55.996802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:55.996706    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:03:56.222291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.187204    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-credential-socket\") pod \"2dc72acb-c794-4eb1-aff0-f23697179813\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") "
Nov 02 14:03:56.222596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.208020    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-credential-socket" pod "2dc72acb-c794-4eb1-aff0-f23697179813" (UID: "2dc72acb-c794-4eb1-aff0-f23697179813"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:56.222772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.222606    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-envoy\") pod \"2dc72acb-c794-4eb1-aff0-f23697179813\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") "
Nov 02 14:03:56.222934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.222910    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-kube-api-access-t2zq8\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-kube-api-access-t2zq8\") pod \"2dc72acb-c794-4eb1-aff0-f23697179813\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") "
Nov 02 14:03:56.223012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.222998    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-certs\") pod \"2dc72acb-c794-4eb1-aff0-f23697179813\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") "
Nov 02 14:03:56.223078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.223053    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-socket\") pod \"2dc72acb-c794-4eb1-aff0-f23697179813\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") "
Nov 02 14:03:56.223137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.223117    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-istio-token\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-istio-token\") pod \"2dc72acb-c794-4eb1-aff0-f23697179813\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") "
Nov 02 14:03:56.223691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.223207    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dc72acb-c794-4eb1-aff0-f23697179813-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc72acb-c794-4eb1-aff0-f23697179813-istio-podinfo\") pod \"2dc72acb-c794-4eb1-aff0-f23697179813\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") "
Nov 02 14:03:56.223691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.223330    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc72acb-c794-4eb1-aff0-f23697179813-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc72acb-c794-4eb1-aff0-f23697179813-istiod-ca-cert\") pod \"2dc72acb-c794-4eb1-aff0-f23697179813\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") "
Nov 02 14:03:56.223691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.223388    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-data\") pod \"2dc72acb-c794-4eb1-aff0-f23697179813\" (UID: \"2dc72acb-c794-4eb1-aff0-f23697179813\") "
Nov 02 14:03:56.223691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.223426    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-certs" pod "2dc72acb-c794-4eb1-aff0-f23697179813" (UID: "2dc72acb-c794-4eb1-aff0-f23697179813"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:56.224191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.224164    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:56.224282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.224199    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:56.224406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.224384    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-data" pod "2dc72acb-c794-4eb1-aff0-f23697179813" (UID: "2dc72acb-c794-4eb1-aff0-f23697179813"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:56.225072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.225037    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc72acb-c794-4eb1-aff0-f23697179813-istiod-ca-cert" pod "2dc72acb-c794-4eb1-aff0-f23697179813" (UID: "2dc72acb-c794-4eb1-aff0-f23697179813"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:03:56.225447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.225422    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-envoy" pod "2dc72acb-c794-4eb1-aff0-f23697179813" (UID: "2dc72acb-c794-4eb1-aff0-f23697179813"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:56.225714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.225687    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-kube-api-access-t2zq8" pod "2dc72acb-c794-4eb1-aff0-f23697179813" (UID: "2dc72acb-c794-4eb1-aff0-f23697179813"). InnerVolumeSpecName "kube-api-access-t2zq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:56.225831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.225729    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-istio-token" pod "2dc72acb-c794-4eb1-aff0-f23697179813" (UID: "2dc72acb-c794-4eb1-aff0-f23697179813"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:03:56.226456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.226435    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dc72acb-c794-4eb1-aff0-f23697179813-istio-podinfo" pod "2dc72acb-c794-4eb1-aff0-f23697179813" (UID: "2dc72acb-c794-4eb1-aff0-f23697179813"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:03:56.226894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.226870    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-socket" pod "2dc72acb-c794-4eb1-aff0-f23697179813" (UID: "2dc72acb-c794-4eb1-aff0-f23697179813"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:03:56.326094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.326058    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2zq8\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-kube-api-access-t2zq8\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:56.326281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.326098    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:56.326281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.326118    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc72acb-c794-4eb1-aff0-f23697179813-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:56.326281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.326133    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc72acb-c794-4eb1-aff0-f23697179813-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:56.326281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.326145    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc72acb-c794-4eb1-aff0-f23697179813-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:56.326281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.326157    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:56.326281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.326169    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc72acb-c794-4eb1-aff0-f23697179813-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:03:56.709647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.709604    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"
Nov 02 14:03:56.709647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.709626    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" event={"ID":"2dc72acb-c794-4eb1-aff0-f23697179813","Type":"ContainerDied","Data":"2efdd8ccb3f148ba1ec1ff8bfdcfe0a4677c60f759844061f35b330b52b9f5e7"}
Nov 02 14:03:56.710186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.709691    5034 scope.go:122] "RemoveContainer" containerID="eca2a0d221d47b45122059b4b51ef7c9fc3861755c5ce5c9a68fc15750449e39"
Nov 02 14:03:56.784303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.784276    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"]
Nov 02 14:03:56.799487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.799461    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs"]
Nov 02 14:03:56.919105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.919071    5034 patch_prober.go:29] interesting pod/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:03:56.919264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:56.919138    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-981903-openshift-default-7dd867666c-vwgbs" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:03:57.402879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:03:57.402842    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" path="/var/lib/kubelet/pods/2dc72acb-c794-4eb1-aff0-f23697179813/volumes"
Nov 02 14:04:00.593600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.593553    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"]
Nov 02 14:04:00.594653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.594625    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy"
Nov 02 14:04:00.594653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.594651    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy"
Nov 02 14:04:00.594880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.594663    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy"
Nov 02 14:04:00.594880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.594671    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy"
Nov 02 14:04:00.594999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.594882    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy"
Nov 02 14:04:00.594999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.594896    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc72acb-c794-4eb1-aff0-f23697179813" containerName="istio-proxy"
Nov 02 14:04:00.603039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.603006    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.637799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.636678    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"]
Nov 02 14:04:00.651805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.649004    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:04:00.651805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.649178    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:04:00.651805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.651067    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:04:00.666951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.662371    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:04:00.703076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.703023    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14749f85-tkhrm\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.703375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.703349    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14749f85-tkhrm\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.703468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.703388    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmljn\" (UniqueName: \"kubernetes.io/projected/cf95407d-4a33-490f-9c03-6294ba0eb004-kube-api-access-tmljn\") pod \"maas-api-revoke-sub-14749f85-tkhrm\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.806008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.804700    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14749f85-tkhrm\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.806008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.804948    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14749f85-tkhrm\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.806008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.804975    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmljn\" (UniqueName: \"kubernetes.io/projected/cf95407d-4a33-490f-9c03-6294ba0eb004-kube-api-access-tmljn\") pod \"maas-api-revoke-sub-14749f85-tkhrm\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.806008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.805928    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14749f85-tkhrm\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.808582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.808553    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14749f85-tkhrm\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.850620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.850556    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmljn\" (UniqueName: \"kubernetes.io/projected/cf95407d-4a33-490f-9c03-6294ba0eb004-kube-api-access-tmljn\") pod \"maas-api-revoke-sub-14749f85-tkhrm\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:00.935819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:00.935787    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:01.277758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:01.277726    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"]
Nov 02 14:04:01.281958 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:04:01.281911    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf95407d_4a33_490f_9c03_6294ba0eb004.slice/crio-6d936daa688fd5ea2f8f0da09a411552b2c0b14ef69a6e8119e808d11480768e WatchSource:0}: Error finding container 6d936daa688fd5ea2f8f0da09a411552b2c0b14ef69a6e8119e808d11480768e: Status 404 returned error can't find the container with id 6d936daa688fd5ea2f8f0da09a411552b2c0b14ef69a6e8119e808d11480768e
Nov 02 14:04:01.808800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:01.808558    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm" event={"ID":"cf95407d-4a33-490f-9c03-6294ba0eb004","Type":"ContainerStarted","Data":"07dd1faf6dd292d0bbe39428cdf6f0d4be20cf00927434c5cbbd2d51a0745651"}
Nov 02 14:04:01.808800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:01.808628    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm" event={"ID":"cf95407d-4a33-490f-9c03-6294ba0eb004","Type":"ContainerStarted","Data":"6d936daa688fd5ea2f8f0da09a411552b2c0b14ef69a6e8119e808d11480768e"}
Nov 02 14:04:01.854510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:01.854424    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm" podStartSLOduration=1.854411272 podStartE2EDuration="1.854411272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:04:01.852073339 +0000 UTC m=+1232.603377778" watchObservedRunningTime="2026-11-02 14:04:01.854411272 +0000 UTC m=+1232.605715709"
Nov 02 14:04:02.820486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:02.820452    5034 generic.go:356] "Generic (PLEG): container finished" podID="cf95407d-4a33-490f-9c03-6294ba0eb004" containerID="07dd1faf6dd292d0bbe39428cdf6f0d4be20cf00927434c5cbbd2d51a0745651" exitCode=0
Nov 02 14:04:02.820911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:02.820510    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm" event={"ID":"cf95407d-4a33-490f-9c03-6294ba0eb004","Type":"ContainerDied","Data":"07dd1faf6dd292d0bbe39428cdf6f0d4be20cf00927434c5cbbd2d51a0745651"}
Nov 02 14:04:04.007634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.007600    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:04.060131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.060094    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-openshift-ca\") pod \"cf95407d-4a33-490f-9c03-6294ba0eb004\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") "
Nov 02 14:04:04.060337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.060215    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-xks-ca\") pod \"cf95407d-4a33-490f-9c03-6294ba0eb004\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") "
Nov 02 14:04:04.060422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.060372    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf95407d-4a33-490f-9c03-6294ba0eb004-kube-api-access-tmljn\" (UniqueName: \"kubernetes.io/projected/cf95407d-4a33-490f-9c03-6294ba0eb004-kube-api-access-tmljn\") pod \"cf95407d-4a33-490f-9c03-6294ba0eb004\" (UID: \"cf95407d-4a33-490f-9c03-6294ba0eb004\") "
Nov 02 14:04:04.060884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.060859    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-openshift-ca" pod "cf95407d-4a33-490f-9c03-6294ba0eb004" (UID: "cf95407d-4a33-490f-9c03-6294ba0eb004"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:04.061120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.061097    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:04.064839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.064764    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-xks-ca" pod "cf95407d-4a33-490f-9c03-6294ba0eb004" (UID: "cf95407d-4a33-490f-9c03-6294ba0eb004"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:04:04.065056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.065031    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf95407d-4a33-490f-9c03-6294ba0eb004-kube-api-access-tmljn" pod "cf95407d-4a33-490f-9c03-6294ba0eb004" (UID: "cf95407d-4a33-490f-9c03-6294ba0eb004"). InnerVolumeSpecName "kube-api-access-tmljn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:04.162711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.162680    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf95407d-4a33-490f-9c03-6294ba0eb004-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:04.162711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.162707    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmljn\" (UniqueName: \"kubernetes.io/projected/cf95407d-4a33-490f-9c03-6294ba0eb004-kube-api-access-tmljn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:04.844761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.844723    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm" event={"ID":"cf95407d-4a33-490f-9c03-6294ba0eb004","Type":"ContainerDied","Data":"6d936daa688fd5ea2f8f0da09a411552b2c0b14ef69a6e8119e808d11480768e"}
Nov 02 14:04:04.846092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.845143    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d936daa688fd5ea2f8f0da09a411552b2c0b14ef69a6e8119e808d11480768e"
Nov 02 14:04:04.846092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:04.845191    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"
Nov 02 14:04:06.039022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:06.038991    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"]
Nov 02 14:04:06.051444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:06.051418    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ba4ccb4-d82lx"]
Nov 02 14:04:07.396327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:07.396296    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f91fa8-40ca-4847-a4f2-1abd215ac6d4" path="/var/lib/kubelet/pods/f8f91fa8-40ca-4847-a4f2-1abd215ac6d4/volumes"
Nov 02 14:04:09.753962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:09.753920    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"]
Nov 02 14:04:14.754974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:14.754924    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg" podUID="fd627ac5-e1ee-48aa-af50-de344e035e73" containerName="payload-processing" containerID="cri-o://89656b8691ba984de993a4c2d7700899d1b8e96c1d6497f8399af9626c34f89d" gracePeriod=25
Nov 02 14:04:14.993696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:14.993256    5034 generic.go:356] "Generic (PLEG): container finished" podID="fd627ac5-e1ee-48aa-af50-de344e035e73" containerID="89656b8691ba984de993a4c2d7700899d1b8e96c1d6497f8399af9626c34f89d" exitCode=0
Nov 02 14:04:14.993696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:14.993388    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg" event={"ID":"fd627ac5-e1ee-48aa-af50-de344e035e73","Type":"ContainerDied","Data":"89656b8691ba984de993a4c2d7700899d1b8e96c1d6497f8399af9626c34f89d"}
Nov 02 14:04:15.212801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:15.199718    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:04:15.324346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:15.316931    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd627ac5-e1ee-48aa-af50-de344e035e73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd627ac5-e1ee-48aa-af50-de344e035e73-plugins-config-volume\") pod \"fd627ac5-e1ee-48aa-af50-de344e035e73\" (UID: \"fd627ac5-e1ee-48aa-af50-de344e035e73\") "
Nov 02 14:04:15.325173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:15.325137    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd627ac5-e1ee-48aa-af50-de344e035e73-kube-api-access-njwch\" (UniqueName: \"kubernetes.io/projected/fd627ac5-e1ee-48aa-af50-de344e035e73-kube-api-access-njwch\") pod \"fd627ac5-e1ee-48aa-af50-de344e035e73\" (UID: \"fd627ac5-e1ee-48aa-af50-de344e035e73\") "
Nov 02 14:04:15.334538 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:15.334503    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd627ac5-e1ee-48aa-af50-de344e035e73-plugins-config-volume" pod "fd627ac5-e1ee-48aa-af50-de344e035e73" (UID: "fd627ac5-e1ee-48aa-af50-de344e035e73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:15.360166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:15.360133    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd627ac5-e1ee-48aa-af50-de344e035e73-kube-api-access-njwch" pod "fd627ac5-e1ee-48aa-af50-de344e035e73" (UID: "fd627ac5-e1ee-48aa-af50-de344e035e73"). InnerVolumeSpecName "kube-api-access-njwch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:15.428250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:15.428214    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njwch\" (UniqueName: \"kubernetes.io/projected/fd627ac5-e1ee-48aa-af50-de344e035e73-kube-api-access-njwch\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:15.428250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:15.428250    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd627ac5-e1ee-48aa-af50-de344e035e73-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:16.005229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:16.005201    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"
Nov 02 14:04:16.005714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:16.005232    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg" event={"ID":"fd627ac5-e1ee-48aa-af50-de344e035e73","Type":"ContainerDied","Data":"150d879d1667f69ce1b066f264be061f73d33a9a6c484c5dd0cb3b9bba168fb2"}
Nov 02 14:04:16.005714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:16.005295    5034 scope.go:122] "RemoveContainer" containerID="89656b8691ba984de993a4c2d7700899d1b8e96c1d6497f8399af9626c34f89d"
Nov 02 14:04:16.067850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:16.067814    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"]
Nov 02 14:04:16.081824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:16.081772    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-584171-688bc6999-fk9kg"]
Nov 02 14:04:17.396946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:17.396912    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd627ac5-e1ee-48aa-af50-de344e035e73" path="/var/lib/kubelet/pods/fd627ac5-e1ee-48aa-af50-de344e035e73/volumes"
Nov 02 14:04:18.415556 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:04:18.415519    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:04:21.147464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:21.147428    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"]
Nov 02 14:04:21.180793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:21.180752    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02122b53-lc964"]
Nov 02 14:04:21.399588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:21.399516    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff39578a-fe15-4580-8f77-430c4168f96c" path="/var/lib/kubelet/pods/ff39578a-fe15-4580-8f77-430c4168f96c/volumes"
Nov 02 14:04:24.032568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:24.032524    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"]
Nov 02 14:04:24.045535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:24.045504    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-979b04d1-hhlwl"]
Nov 02 14:04:24.686140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:24.686097    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"]
Nov 02 14:04:24.687270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:24.687050    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy" containerID="cri-o://f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04" gracePeriod=30
Nov 02 14:04:25.036704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:25.036670    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"]
Nov 02 14:04:25.043792 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:25.043747    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b961700-rcjpx"]
Nov 02 14:04:25.399257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:25.399170    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badb60d3-8f91-4126-8f5f-9a3dd2238036" path="/var/lib/kubelet/pods/badb60d3-8f91-4126-8f5f-9a3dd2238036/volumes"
Nov 02 14:04:25.399964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:25.399943    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43f5b71-686d-4e91-b0be-43eb9c014584" path="/var/lib/kubelet/pods/e43f5b71-686d-4e91-b0be-43eb9c014584/volumes"
Nov 02 14:04:26.151638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:26.151599    5034 patch_prober.go:29] interesting pod/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:04:26.152221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:26.151662    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:04:30.009931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.009899    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:04:30.181258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.181215    5034 generic.go:356] "Generic (PLEG): container finished" podID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerID="f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04" exitCode=0
Nov 02 14:04:30.181461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.181265    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" event={"ID":"dd6328bd-5689-4d4e-9fa0-3ab417072485","Type":"ContainerDied","Data":"f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04"}
Nov 02 14:04:30.181461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.181302    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"
Nov 02 14:04:30.181461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.181361    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z" event={"ID":"dd6328bd-5689-4d4e-9fa0-3ab417072485","Type":"ContainerDied","Data":"c8660fbf340057f54f8c45d49961f61d64077a39d86821b41a31e2700b4e6d1d"}
Nov 02 14:04:30.181461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.181402    5034 scope.go:122] "RemoveContainer" containerID="f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04"
Nov 02 14:04:30.193236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.193209    5034 scope.go:122] "RemoveContainer" containerID="e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597"
Nov 02 14:04:30.203754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.203728    5034 scope.go:122] "RemoveContainer" containerID="f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04"
Nov 02 14:04:30.204300 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:04:30.204279    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04\": container with ID starting with f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04 not found: ID does not exist" containerID="f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04"
Nov 02 14:04:30.204361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.204311    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04"} err="failed to get container status \"f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04\": rpc error: code = NotFound desc = could not find container \"f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04\": container with ID starting with f675751f2edf924719646bb987028b80508a72faa8c097e114c1442d17fe9e04 not found: ID does not exist"
Nov 02 14:04:30.204361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.204329    5034 scope.go:122] "RemoveContainer" containerID="e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597"
Nov 02 14:04:30.204585 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:04:30.204565    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597\": container with ID starting with e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597 not found: ID does not exist" containerID="e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597"
Nov 02 14:04:30.204642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.204594    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597"} err="failed to get container status \"e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597\": rpc error: code = NotFound desc = could not find container \"e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597\": container with ID starting with e3d3c788d9cd3b28caaafa85b391c066785dfa1eed24c8502800951c8a870597 not found: ID does not exist"
Nov 02 14:04:30.206034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.205997    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-credential-socket\") pod \"dd6328bd-5689-4d4e-9fa0-3ab417072485\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") "
Nov 02 14:04:30.206157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206048    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd6328bd-5689-4d4e-9fa0-3ab417072485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6328bd-5689-4d4e-9fa0-3ab417072485-istiod-ca-cert\") pod \"dd6328bd-5689-4d4e-9fa0-3ab417072485\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") "
Nov 02 14:04:30.206157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206116    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-kube-api-access-zvcvz\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-kube-api-access-zvcvz\") pod \"dd6328bd-5689-4d4e-9fa0-3ab417072485\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") "
Nov 02 14:04:30.206282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206185    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-data\") pod \"dd6328bd-5689-4d4e-9fa0-3ab417072485\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") "
Nov 02 14:04:30.206282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206222    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-podinfo\") pod \"dd6328bd-5689-4d4e-9fa0-3ab417072485\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") "
Nov 02 14:04:30.206383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206288    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-credential-socket" pod "dd6328bd-5689-4d4e-9fa0-3ab417072485" (UID: "dd6328bd-5689-4d4e-9fa0-3ab417072485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:04:30.206383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206336    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-socket\") pod \"dd6328bd-5689-4d4e-9fa0-3ab417072485\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") "
Nov 02 14:04:30.206495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206396    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-envoy\") pod \"dd6328bd-5689-4d4e-9fa0-3ab417072485\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") "
Nov 02 14:04:30.206495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206443    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-token\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-token\") pod \"dd6328bd-5689-4d4e-9fa0-3ab417072485\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") "
Nov 02 14:04:30.206601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206489    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-data" pod "dd6328bd-5689-4d4e-9fa0-3ab417072485" (UID: "dd6328bd-5689-4d4e-9fa0-3ab417072485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:04:30.206601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206513    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd6328bd-5689-4d4e-9fa0-3ab417072485-istiod-ca-cert" pod "dd6328bd-5689-4d4e-9fa0-3ab417072485" (UID: "dd6328bd-5689-4d4e-9fa0-3ab417072485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:30.206601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206548    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-certs\") pod \"dd6328bd-5689-4d4e-9fa0-3ab417072485\" (UID: \"dd6328bd-5689-4d4e-9fa0-3ab417072485\") "
Nov 02 14:04:30.206723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206600    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-socket" pod "dd6328bd-5689-4d4e-9fa0-3ab417072485" (UID: "dd6328bd-5689-4d4e-9fa0-3ab417072485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:04:30.206829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.206767    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-certs" pod "dd6328bd-5689-4d4e-9fa0-3ab417072485" (UID: "dd6328bd-5689-4d4e-9fa0-3ab417072485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:04:30.207352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.207332    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:30.207352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.207352    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:30.207502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.207361    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:30.207502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.207375    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:30.207502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.207384    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd6328bd-5689-4d4e-9fa0-3ab417072485-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:30.208680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.208654    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-envoy" pod "dd6328bd-5689-4d4e-9fa0-3ab417072485" (UID: "dd6328bd-5689-4d4e-9fa0-3ab417072485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:04:30.208905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.208881    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-token" pod "dd6328bd-5689-4d4e-9fa0-3ab417072485" (UID: "dd6328bd-5689-4d4e-9fa0-3ab417072485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:30.208999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.208976    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-podinfo" pod "dd6328bd-5689-4d4e-9fa0-3ab417072485" (UID: "dd6328bd-5689-4d4e-9fa0-3ab417072485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:04:30.209161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.209139    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-kube-api-access-zvcvz" pod "dd6328bd-5689-4d4e-9fa0-3ab417072485" (UID: "dd6328bd-5689-4d4e-9fa0-3ab417072485"). InnerVolumeSpecName "kube-api-access-zvcvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:30.309283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.309229    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvcvz\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-kube-api-access-zvcvz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:30.309283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.309276    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:30.309283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.309287    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:30.309283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.309295    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd6328bd-5689-4d4e-9fa0-3ab417072485-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:30.550168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.550134    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"]
Nov 02 14:04:30.560542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:30.560510    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-584171-openshift-default-57cdf6487c-tln9z"]
Nov 02 14:04:31.397167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:31.397133    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" path="/var/lib/kubelet/pods/dd6328bd-5689-4d4e-9fa0-3ab417072485/volumes"
Nov 02 14:04:31.849692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:31.849650    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"]
Nov 02 14:04:31.877543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:31.877508    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"]
Nov 02 14:04:34.797853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:34.797816    5034 scope.go:122] "RemoveContainer" containerID="b78230db0456b0d3719cf3d8976a58d73838302f2caa68ba6026639d4bbd34b9"
Nov 02 14:04:36.850967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:36.850927    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c" podUID="38608c61-0ac5-41fe-b2eb-04f7f83c387f" containerName="payload-processing" containerID="cri-o://cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987" gracePeriod=25
Nov 02 14:04:36.878226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:36.878178    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx" podUID="fadceb97-3b8e-49a1-99fc-0324f0c50c63" containerName="payload-pre-processing" containerID="cri-o://a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10" gracePeriod=25
Nov 02 14:04:37.202366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.202332    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:04:37.238049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.238022    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38608c61-0ac5-41fe-b2eb-04f7f83c387f-kube-api-access-k62cg\" (UniqueName: \"kubernetes.io/projected/38608c61-0ac5-41fe-b2eb-04f7f83c387f-kube-api-access-k62cg\") pod \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\" (UID: \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\") "
Nov 02 14:04:37.238275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.238259    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38608c61-0ac5-41fe-b2eb-04f7f83c387f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38608c61-0ac5-41fe-b2eb-04f7f83c387f-plugins-config-volume\") pod \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\" (UID: \"38608c61-0ac5-41fe-b2eb-04f7f83c387f\") "
Nov 02 14:04:37.238600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.238577    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38608c61-0ac5-41fe-b2eb-04f7f83c387f-plugins-config-volume" pod "38608c61-0ac5-41fe-b2eb-04f7f83c387f" (UID: "38608c61-0ac5-41fe-b2eb-04f7f83c387f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:37.238872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.238853    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38608c61-0ac5-41fe-b2eb-04f7f83c387f-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:37.240138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.240120    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38608c61-0ac5-41fe-b2eb-04f7f83c387f-kube-api-access-k62cg" pod "38608c61-0ac5-41fe-b2eb-04f7f83c387f" (UID: "38608c61-0ac5-41fe-b2eb-04f7f83c387f"). InnerVolumeSpecName "kube-api-access-k62cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:37.253057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.253034    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:04:37.263281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.263252    5034 generic.go:356] "Generic (PLEG): container finished" podID="fadceb97-3b8e-49a1-99fc-0324f0c50c63" containerID="a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10" exitCode=0
Nov 02 14:04:37.263397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.263354    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"
Nov 02 14:04:37.263468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.263389    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx" event={"ID":"fadceb97-3b8e-49a1-99fc-0324f0c50c63","Type":"ContainerDied","Data":"a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10"}
Nov 02 14:04:37.263468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.263453    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx" event={"ID":"fadceb97-3b8e-49a1-99fc-0324f0c50c63","Type":"ContainerDied","Data":"269259489f773817844db96946a8832d7a765a4af9c543c3b7519878b2876d07"}
Nov 02 14:04:37.263597 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.263482    5034 scope.go:122] "RemoveContainer" containerID="a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10"
Nov 02 14:04:37.265322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.265294    5034 generic.go:356] "Generic (PLEG): container finished" podID="38608c61-0ac5-41fe-b2eb-04f7f83c387f" containerID="cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987" exitCode=0
Nov 02 14:04:37.265600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.265579    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c" event={"ID":"38608c61-0ac5-41fe-b2eb-04f7f83c387f","Type":"ContainerDied","Data":"cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987"}
Nov 02 14:04:37.265693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.265630    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c" event={"ID":"38608c61-0ac5-41fe-b2eb-04f7f83c387f","Type":"ContainerDied","Data":"230e406737ed921b5f38851e3644cefa51fbff2539c7f71842c38c1af89af213"}
Nov 02 14:04:37.265757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.265707    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"
Nov 02 14:04:37.280955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.280919    5034 scope.go:122] "RemoveContainer" containerID="a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10"
Nov 02 14:04:37.281261 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:04:37.281231    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10\": container with ID starting with a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10 not found: ID does not exist" containerID="a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10"
Nov 02 14:04:37.281379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.281277    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10"} err="failed to get container status \"a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10\": rpc error: code = NotFound desc = could not find container \"a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10\": container with ID starting with a56bba8cad81bc4c6414ba5cfcc0f4689fe40023e89fec46614b7dfbc2fe2f10 not found: ID does not exist"
Nov 02 14:04:37.281379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.281300    5034 scope.go:122] "RemoveContainer" containerID="cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987"
Nov 02 14:04:37.292838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.292817    5034 scope.go:122] "RemoveContainer" containerID="cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987"
Nov 02 14:04:37.293090 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:04:37.293063    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987\": container with ID starting with cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987 not found: ID does not exist" containerID="cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987"
Nov 02 14:04:37.293168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.293087    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987"} err="failed to get container status \"cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987\": rpc error: code = NotFound desc = could not find container \"cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987\": container with ID starting with cfa55923f8c4c1ea77220ee40c4133c19279076b7574ec94969ed2ddd3a61987 not found: ID does not exist"
Nov 02 14:04:37.340395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.340367    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadceb97-3b8e-49a1-99fc-0324f0c50c63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadceb97-3b8e-49a1-99fc-0324f0c50c63-plugins-config-volume\") pod \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\" (UID: \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\") "
Nov 02 14:04:37.340572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.340450    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadceb97-3b8e-49a1-99fc-0324f0c50c63-kube-api-access-6wjtc\" (UniqueName: \"kubernetes.io/projected/fadceb97-3b8e-49a1-99fc-0324f0c50c63-kube-api-access-6wjtc\") pod \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\" (UID: \"fadceb97-3b8e-49a1-99fc-0324f0c50c63\") "
Nov 02 14:04:37.340824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.340772    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadceb97-3b8e-49a1-99fc-0324f0c50c63-plugins-config-volume" pod "fadceb97-3b8e-49a1-99fc-0324f0c50c63" (UID: "fadceb97-3b8e-49a1-99fc-0324f0c50c63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:37.341066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.341046    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62cg\" (UniqueName: \"kubernetes.io/projected/38608c61-0ac5-41fe-b2eb-04f7f83c387f-kube-api-access-k62cg\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:37.341066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.341064    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fadceb97-3b8e-49a1-99fc-0324f0c50c63-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:37.343258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.343237    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadceb97-3b8e-49a1-99fc-0324f0c50c63-kube-api-access-6wjtc" pod "fadceb97-3b8e-49a1-99fc-0324f0c50c63" (UID: "fadceb97-3b8e-49a1-99fc-0324f0c50c63"). InnerVolumeSpecName "kube-api-access-6wjtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:37.366970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.366943    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"]
Nov 02 14:04:37.372627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.372601    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c"]
Nov 02 14:04:37.397677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.397614    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38608c61-0ac5-41fe-b2eb-04f7f83c387f" path="/var/lib/kubelet/pods/38608c61-0ac5-41fe-b2eb-04f7f83c387f/volumes"
Nov 02 14:04:37.442185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.442157    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wjtc\" (UniqueName: \"kubernetes.io/projected/fadceb97-3b8e-49a1-99fc-0324f0c50c63-kube-api-access-6wjtc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:37.609599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.609569    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"]
Nov 02 14:04:37.616554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:37.616518    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8d5eec-79694d6495-m7jqx"]
Nov 02 14:04:38.100406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:38.100371    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-8d5eec-5c7db55b6f-frp2c" podUID="38608c61-0ac5-41fe-b2eb-04f7f83c387f" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.151:50052\" within 1s: context deadline exceeded"
Nov 02 14:04:38.100757 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:04:38.100446    5034 logging.go:55] [core] [Channel #1551 SubChannel #1552]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.151:50052", ServerName: "10.132.0.151:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.151:50052: operation was canceled"
Nov 02 14:04:39.030788 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:39.030744    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"]
Nov 02 14:04:39.043956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:39.043926    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62d126b4-tt5gk"]
Nov 02 14:04:39.398912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:39.398826    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a" path="/var/lib/kubelet/pods/42f2f1cf-d0ce-4c0f-ba95-8a1898374b6a/volumes"
Nov 02 14:04:39.399434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:39.399413    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadceb97-3b8e-49a1-99fc-0324f0c50c63" path="/var/lib/kubelet/pods/fadceb97-3b8e-49a1-99fc-0324f0c50c63/volumes"
Nov 02 14:04:40.031195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:40.031162    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"]
Nov 02 14:04:40.044030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:40.044002    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c54e268-wnsk9"]
Nov 02 14:04:41.396018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:41.395975    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aea1ff31-bc92-4d49-82ba-18924cbe8664" path="/var/lib/kubelet/pods/aea1ff31-bc92-4d49-82ba-18924cbe8664/volumes"
Nov 02 14:04:45.032529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:45.032496    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"]
Nov 02 14:04:45.045292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:45.045258    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d62804e-mqnm9"]
Nov 02 14:04:45.396661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:45.396567    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5dc46cd-79c7-4598-9ede-b2c594efc915" path="/var/lib/kubelet/pods/b5dc46cd-79c7-4598-9ede-b2c594efc915/volumes"
Nov 02 14:04:48.436169 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:04:48.436123    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:04:49.914911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:49.914870    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"]
Nov 02 14:04:50.053698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:50.053662    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"]
Nov 02 14:04:50.057680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:50.057653    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fcc369c-c99vx"]
Nov 02 14:04:51.166467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:51.166434    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"]
Nov 02 14:04:51.196193 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:51.196158    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9953549-9wn8s"]
Nov 02 14:04:51.397960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:51.397929    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90beea4a-ca68-454f-95c9-d7576b8d8c8d" path="/var/lib/kubelet/pods/90beea4a-ca68-454f-95c9-d7576b8d8c8d/volumes"
Nov 02 14:04:51.398488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:51.398471    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9366740-9c2a-4df2-b1b1-e11f7beb2511" path="/var/lib/kubelet/pods/b9366740-9c2a-4df2-b1b1-e11f7beb2511/volumes"
Nov 02 14:04:54.916236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:54.916194    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2" podUID="5f386e34-64f2-4f82-876a-b40c0287ad32" containerName="payload-processing" containerID="cri-o://13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b" gracePeriod=25
Nov 02 14:04:55.245647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.245616    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:04:55.368425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.368387    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f386e34-64f2-4f82-876a-b40c0287ad32-kube-api-access-rp2mh\" (UniqueName: \"kubernetes.io/projected/5f386e34-64f2-4f82-876a-b40c0287ad32-kube-api-access-rp2mh\") pod \"5f386e34-64f2-4f82-876a-b40c0287ad32\" (UID: \"5f386e34-64f2-4f82-876a-b40c0287ad32\") "
Nov 02 14:04:55.368602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.368455    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f386e34-64f2-4f82-876a-b40c0287ad32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f386e34-64f2-4f82-876a-b40c0287ad32-plugins-config-volume\") pod \"5f386e34-64f2-4f82-876a-b40c0287ad32\" (UID: \"5f386e34-64f2-4f82-876a-b40c0287ad32\") "
Nov 02 14:04:55.369165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.369114    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f386e34-64f2-4f82-876a-b40c0287ad32-plugins-config-volume" pod "5f386e34-64f2-4f82-876a-b40c0287ad32" (UID: "5f386e34-64f2-4f82-876a-b40c0287ad32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:55.369528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.369500    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f386e34-64f2-4f82-876a-b40c0287ad32-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:55.388839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.387631    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f386e34-64f2-4f82-876a-b40c0287ad32-kube-api-access-rp2mh" pod "5f386e34-64f2-4f82-876a-b40c0287ad32" (UID: "5f386e34-64f2-4f82-876a-b40c0287ad32"). InnerVolumeSpecName "kube-api-access-rp2mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:55.470735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.470661    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp2mh\" (UniqueName: \"kubernetes.io/projected/5f386e34-64f2-4f82-876a-b40c0287ad32-kube-api-access-rp2mh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:55.636961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.636861    5034 generic.go:356] "Generic (PLEG): container finished" podID="5f386e34-64f2-4f82-876a-b40c0287ad32" containerID="13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b" exitCode=0
Nov 02 14:04:55.636961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.636963    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2" event={"ID":"5f386e34-64f2-4f82-876a-b40c0287ad32","Type":"ContainerDied","Data":"13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b"}
Nov 02 14:04:55.637294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.637016    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2" event={"ID":"5f386e34-64f2-4f82-876a-b40c0287ad32","Type":"ContainerDied","Data":"702621e950dc45ad519ba8e9cd22d2da9ca19a78f7dc67994bb6f2b33631bc2d"}
Nov 02 14:04:55.637294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.637047    5034 scope.go:122] "RemoveContainer" containerID="13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b"
Nov 02 14:04:55.637294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.637221    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"
Nov 02 14:04:55.651996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.651973    5034 scope.go:122] "RemoveContainer" containerID="13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b"
Nov 02 14:04:55.652356 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:04:55.652330    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b\": container with ID starting with 13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b not found: ID does not exist" containerID="13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b"
Nov 02 14:04:55.652442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.652366    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b"} err="failed to get container status \"13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b\": rpc error: code = NotFound desc = could not find container \"13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b\": container with ID starting with 13bff8079df2fe3348fdf2b291185d47be18d618e6968d4e333b6293c8fcda3b not found: ID does not exist"
Nov 02 14:04:55.738460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.738416    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"]
Nov 02 14:04:55.750049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:55.750023    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d776122a-67c8cdd4cf-lvdn2"]
Nov 02 14:04:57.033576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:57.033542    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"]
Nov 02 14:04:57.047866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:57.047835    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf295fdf-x8zkx"]
Nov 02 14:04:57.396192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:57.396107    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f386e34-64f2-4f82-876a-b40c0287ad32" path="/var/lib/kubelet/pods/5f386e34-64f2-4f82-876a-b40c0287ad32/volumes"
Nov 02 14:04:57.396587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:57.396572    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b259e13e-1592-4c14-9914-c3c60a29bc3f" path="/var/lib/kubelet/pods/b259e13e-1592-4c14-9914-c3c60a29bc3f/volumes"
Nov 02 14:04:57.657576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:57.657494    5034 generic.go:356] "Generic (PLEG): container finished" podID="cadb576c-9eeb-4d28-8afb-5ac016a0bbc9" containerID="99ef4b9c07185b93095bb55d7c243e9e305fc0d12a0a49045e7943fa035f29f7" exitCode=0
Nov 02 14:04:57.657724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:57.657577    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f2hmh-rrqpl" event={"ID":"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9","Type":"ContainerDied","Data":"99ef4b9c07185b93095bb55d7c243e9e305fc0d12a0a49045e7943fa035f29f7"}
Nov 02 14:04:58.860865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.860842    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:04:58.957108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.954758    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-archives-path\") pod \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") "
Nov 02 14:04:58.957108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.955039    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-service-ca-bundle\") pod \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") "
Nov 02 14:04:58.957108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.955153    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-kube-api-access-jsg4m\" (UniqueName: \"kubernetes.io/projected/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-kube-api-access-jsg4m\") pod \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") "
Nov 02 14:04:58.957108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.955205    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-trusted-ca-bundle\") pod \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\" (UID: \"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9\") "
Nov 02 14:04:58.957108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.956151    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-archives-path" pod "cadb576c-9eeb-4d28-8afb-5ac016a0bbc9" (UID: "cadb576c-9eeb-4d28-8afb-5ac016a0bbc9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:04:58.958338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.958303    5034 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-archives-path\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:58.959648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.958946    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-service-ca-bundle" pod "cadb576c-9eeb-4d28-8afb-5ac016a0bbc9" (UID: "cadb576c-9eeb-4d28-8afb-5ac016a0bbc9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:58.962731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.962689    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-trusted-ca-bundle" pod "cadb576c-9eeb-4d28-8afb-5ac016a0bbc9" (UID: "cadb576c-9eeb-4d28-8afb-5ac016a0bbc9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:04:58.970731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:58.968810    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-kube-api-access-jsg4m" pod "cadb576c-9eeb-4d28-8afb-5ac016a0bbc9" (UID: "cadb576c-9eeb-4d28-8afb-5ac016a0bbc9"). InnerVolumeSpecName "kube-api-access-jsg4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:04:59.059581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:59.059549    5034 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-service-ca-bundle\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:59.059581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:59.059579    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsg4m\" (UniqueName: \"kubernetes.io/projected/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-kube-api-access-jsg4m\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:59.059581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:59.059589    5034 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cadb576c-9eeb-4d28-8afb-5ac016a0bbc9-trusted-ca-bundle\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:04:59.693276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:59.693199    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-f2hmh-rrqpl" event={"ID":"cadb576c-9eeb-4d28-8afb-5ac016a0bbc9","Type":"ContainerDied","Data":"b3eab483400d6ce0c381ec011e98dffefa2105e9629a35a67fdc403d1e5482d4"}
Nov 02 14:04:59.693468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:59.693457    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3eab483400d6ce0c381ec011e98dffefa2105e9629a35a67fdc403d1e5482d4"
Nov 02 14:04:59.693600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:04:59.693590    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-f2hmh-rrqpl"
Nov 02 14:05:15.049099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:15.049015    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"]
Nov 02 14:05:15.062468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:15.062431    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa9b9d92-9qw5w"]
Nov 02 14:05:15.399817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:15.399717    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8643d5d-ca1c-4c95-9281-5f1e48b77ddd" path="/var/lib/kubelet/pods/e8643d5d-ca1c-4c95-9281-5f1e48b77ddd/volumes"
Nov 02 14:05:18.417021 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:05:18.416986    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:05:24.728108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728068    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"]
Nov 02 14:05:24.728735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728718    5034 cpu_manager.go:414] "containerMap: removing container" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy"
Nov 02 14:05:24.728735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728733    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728747    5034 cpu_manager.go:414] "containerMap: removing container" podUID="fd627ac5-e1ee-48aa-af50-de344e035e73" containerName="payload-processing"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728752    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd627ac5-e1ee-48aa-af50-de344e035e73" containerName="payload-processing"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728800    5034 cpu_manager.go:414] "containerMap: removing container" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728810    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728826    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5f386e34-64f2-4f82-876a-b40c0287ad32" containerName="payload-processing"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728833    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f386e34-64f2-4f82-876a-b40c0287ad32" containerName="payload-processing"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728840    5034 cpu_manager.go:414] "containerMap: removing container" podUID="38608c61-0ac5-41fe-b2eb-04f7f83c387f" containerName="payload-processing"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728844    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38608c61-0ac5-41fe-b2eb-04f7f83c387f" containerName="payload-processing"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728853    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cadb576c-9eeb-4d28-8afb-5ac016a0bbc9" containerName="insights-gathering"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728857    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadb576c-9eeb-4d28-8afb-5ac016a0bbc9" containerName="insights-gathering"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728866    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cf95407d-4a33-490f-9c03-6294ba0eb004" containerName="revoke-keys"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728871    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf95407d-4a33-490f-9c03-6294ba0eb004" containerName="revoke-keys"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728879    5034 cpu_manager.go:414] "containerMap: removing container" podUID="fadceb97-3b8e-49a1-99fc-0324f0c50c63" containerName="payload-pre-processing"
Nov 02 14:05:24.728953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728884    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fadceb97-3b8e-49a1-99fc-0324f0c50c63" containerName="payload-pre-processing"
Nov 02 14:05:24.729641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728979    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f386e34-64f2-4f82-876a-b40c0287ad32" containerName="payload-processing"
Nov 02 14:05:24.729641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728989    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="38608c61-0ac5-41fe-b2eb-04f7f83c387f" containerName="payload-processing"
Nov 02 14:05:24.729641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.728997    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadb576c-9eeb-4d28-8afb-5ac016a0bbc9" containerName="insights-gathering"
Nov 02 14:05:24.729641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.729003    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf95407d-4a33-490f-9c03-6294ba0eb004" containerName="revoke-keys"
Nov 02 14:05:24.729641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.729010    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="fadceb97-3b8e-49a1-99fc-0324f0c50c63" containerName="payload-pre-processing"
Nov 02 14:05:24.729641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.729023    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy"
Nov 02 14:05:24.729641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.729029    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd627ac5-e1ee-48aa-af50-de344e035e73" containerName="payload-processing"
Nov 02 14:05:24.729641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.729035    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd6328bd-5689-4d4e-9fa0-3ab417072485" containerName="istio-proxy"
Nov 02 14:05:24.731641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.731612    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:24.752110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.752082    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"]
Nov 02 14:05:24.761342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.761321    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c7d2df-dockercfg-b9s8n\""
Nov 02 14:05:24.794844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.794818    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c7d2df\""
Nov 02 14:05:24.870955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.870912    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgsgp\" (UniqueName: \"kubernetes.io/projected/0f450776-6c08-461b-866b-aa2ade15fabd-kube-api-access-xgsgp\") pod \"payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc\" (UID: \"0f450776-6c08-461b-866b-aa2ade15fabd\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:24.871388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.871360    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f450776-6c08-461b-866b-aa2ade15fabd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc\" (UID: \"0f450776-6c08-461b-866b-aa2ade15fabd\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:24.975942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.975896    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f450776-6c08-461b-866b-aa2ade15fabd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc\" (UID: \"0f450776-6c08-461b-866b-aa2ade15fabd\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:24.978889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.978337    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f450776-6c08-461b-866b-aa2ade15fabd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc\" (UID: \"0f450776-6c08-461b-866b-aa2ade15fabd\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:24.979087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:24.978766    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgsgp\" (UniqueName: \"kubernetes.io/projected/0f450776-6c08-461b-866b-aa2ade15fabd-kube-api-access-xgsgp\") pod \"payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc\" (UID: \"0f450776-6c08-461b-866b-aa2ade15fabd\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:25.003970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:25.003939    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgsgp\" (UniqueName: \"kubernetes.io/projected/0f450776-6c08-461b-866b-aa2ade15fabd-kube-api-access-xgsgp\") pod \"payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc\" (UID: \"0f450776-6c08-461b-866b-aa2ade15fabd\") " pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:25.046685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:25.046657    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:25.448865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:25.448832    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"]
Nov 02 14:05:26.002657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:26.002613    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc" event={"ID":"0f450776-6c08-461b-866b-aa2ade15fabd","Type":"ContainerStarted","Data":"4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f"}
Nov 02 14:05:26.003300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:26.002673    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc" event={"ID":"0f450776-6c08-461b-866b-aa2ade15fabd","Type":"ContainerStarted","Data":"3bcb6f16928a170f23c6500f6dd54faa775ba0d0c2e9bf1147065be03716bdd2"}
Nov 02 14:05:26.003300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:26.002821    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:26.046971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:26.046924    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc" podStartSLOduration=2.046910497 podStartE2EDuration="2.046910497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:26.045097065 +0000 UTC m=+1316.796401503" watchObservedRunningTime="2026-11-02 14:05:26.046910497 +0000 UTC m=+1316.798214935"
Nov 02 14:05:27.310070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.310034    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"]
Nov 02 14:05:27.315056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.315023    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.332299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.332272    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fb206f-openshift-default-dockercfg-zrf9d\""
Nov 02 14:05:27.334932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.334906    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-credential-socket\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.335113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.334973    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrvrx\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-kube-api-access-hrvrx\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.335113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.335011    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-certs\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.335287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.335261    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9109731a-74f9-4294-92f3-dc6da573a3c8-istiod-ca-cert\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.335366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.335307    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-podinfo\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.335366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.335341    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-socket\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.335469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.335396    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-token\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.335522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.335478    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-data\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.335580 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.335536    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-envoy\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.350307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.350276    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"]
Nov 02 14:05:27.436435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.436409    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9109731a-74f9-4294-92f3-dc6da573a3c8-istiod-ca-cert\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.436656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.436473    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-podinfo\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.436656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.436508    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-socket\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.436656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.436555    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-token\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.436656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.436630    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-data\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.436941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.436698    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-envoy\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.436941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.436800    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-credential-socket\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.436941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.436851    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrvrx\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-kube-api-access-hrvrx\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.436941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.436882    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-certs\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.437191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.437057    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-data\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.437253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.437198    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9109731a-74f9-4294-92f3-dc6da573a3c8-istiod-ca-cert\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.437311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.437248    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-socket\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.437390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.437361    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-credential-socket\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.437513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.437403    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-certs\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.439318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.439301    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-envoy\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.439400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.439364    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-podinfo\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.462611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.462586    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-token\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.462920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.462898    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrvrx\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-kube-api-access-hrvrx\") pod \"e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") " pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.639030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.638946    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:27.994040 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:05:27.993052    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9109731a_74f9_4294_92f3_dc6da573a3c8.slice/crio-ff8127e4d50f2c6f1065941b8732ca84983b6407a3e2c5786360957beecd7188 WatchSource:0}: Error finding container ff8127e4d50f2c6f1065941b8732ca84983b6407a3e2c5786360957beecd7188: Status 404 returned error can't find the container with id ff8127e4d50f2c6f1065941b8732ca84983b6407a3e2c5786360957beecd7188
Nov 02 14:05:27.996232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.996204    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"]
Nov 02 14:05:27.996541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.996516    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:05:27.996582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.996573    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:05:27.996621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:27.996602    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:05:28.025829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:28.025805    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" event={"ID":"9109731a-74f9-4294-92f3-dc6da573a3c8","Type":"ContainerStarted","Data":"ff8127e4d50f2c6f1065941b8732ca84983b6407a3e2c5786360957beecd7188"}
Nov 02 14:05:29.034929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:29.034897    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" event={"ID":"9109731a-74f9-4294-92f3-dc6da573a3c8","Type":"ContainerStarted","Data":"a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de"}
Nov 02 14:05:29.098226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:29.098170    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" podStartSLOduration=2.098156567 podStartE2EDuration="2.098156567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:29.095868521 +0000 UTC m=+1319.847172960" watchObservedRunningTime="2026-11-02 14:05:29.098156567 +0000 UTC m=+1319.849461004"
Nov 02 14:05:29.640000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:29.639967    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:29.645513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:29.645487    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:30.043293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:30.043262    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:30.044257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:30.044235    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:05:34.858058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:34.858031    5034 scope.go:122] "RemoveContainer" containerID="3ebe64f48cb7792f7ed878a50eb334faa40bae86627da2f5589533e6bef9af87"
Nov 02 14:05:34.868495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:34.868466    5034 scope.go:122] "RemoveContainer" containerID="2121ea35e202aeeed604bf63787c95326a0aac9f5e16747042f4417553d59521"
Nov 02 14:05:34.880532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:34.880510    5034 scope.go:122] "RemoveContainer" containerID="e58c7470acaad739a09170aae9ba32e35d3a448bc2fe243b7b5456fa5ae3ea8b"
Nov 02 14:05:34.891138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:34.891119    5034 scope.go:122] "RemoveContainer" containerID="7e23aa254c0f17ee7858ec55a3606367742dddc8e227c43964761eda0c8c377b"
Nov 02 14:05:37.016026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:37.016000    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:05:38.035462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:38.035427    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"]
Nov 02 14:05:38.049041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:38.049011    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"]
Nov 02 14:05:38.052686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:38.052662    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-463bdbfb-6w7fk"]
Nov 02 14:05:38.055953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:38.055930    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dea321a-fhjq4"]
Nov 02 14:05:39.397509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:39.397472    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4419c359-907e-4270-92a2-58980835b4de" path="/var/lib/kubelet/pods/4419c359-907e-4270-92a2-58980835b4de/volumes"
Nov 02 14:05:39.397989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:39.397957    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2bcce5-b059-48ec-8fd4-9eb78afa1d70" path="/var/lib/kubelet/pods/7d2bcce5-b059-48ec-8fd4-9eb78afa1d70/volumes"
Nov 02 14:05:41.589039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.589006    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"]
Nov 02 14:05:41.599174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.599146    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.616472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.616440    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"]
Nov 02 14:05:41.624434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.624408    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-60ad62-openshift-default-dockercfg-6tnvl\""
Nov 02 14:05:41.696041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.696010    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-envoy\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.696212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.696162    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-data\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.696257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.696234    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-socket\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.696310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.696299    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae86fff-9730-4845-b60e-a0eac4259ef8-istiod-ca-cert\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.696412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.696397    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-podinfo\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.696457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.696422    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-credential-socket\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.696544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.696528    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-certs\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.696649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.696633    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhzx\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-kube-api-access-gnhzx\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.696699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.696684    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-token\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.807906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.807864    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-podinfo\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.808188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.808163    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-credential-socket\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.808363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.808346    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-certs\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.808556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.808539    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhzx\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-kube-api-access-gnhzx\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.808695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.808679    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-token\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.808893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.808878    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-envoy\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.809080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.809068    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-data\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.809175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.809163    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-socket\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.809277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.809265    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae86fff-9730-4845-b60e-a0eac4259ef8-istiod-ca-cert\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.816803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.810164    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae86fff-9730-4845-b60e-a0eac4259ef8-istiod-ca-cert\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.833805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.830226    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-credential-socket\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.833805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.830557    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-data\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.833805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.830997    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-socket\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.833805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.831488    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-certs\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.834726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.834701    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-podinfo\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.847863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.843209    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-envoy\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.851591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.851565    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-token\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.864807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.857324    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhzx\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-kube-api-access-gnhzx\") pod \"e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") " pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:41.940429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:41.940127    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:42.323873 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:05:42.323843    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae86fff_9730_4845_b60e_a0eac4259ef8.slice/crio-9aba97f817f1d3ffa1a2729b38f8acd7087af4e62e91ace7182afa0e1c7bd106 WatchSource:0}: Error finding container 9aba97f817f1d3ffa1a2729b38f8acd7087af4e62e91ace7182afa0e1c7bd106: Status 404 returned error can't find the container with id 9aba97f817f1d3ffa1a2729b38f8acd7087af4e62e91ace7182afa0e1c7bd106
Nov 02 14:05:42.327117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:42.327073    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:05:42.327236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:42.327164    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:05:42.327291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:42.327243    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:05:42.327564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:42.327541    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"]
Nov 02 14:05:43.033436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:43.033406    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"]
Nov 02 14:05:43.046540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:43.046511    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79d8234d-577w5"]
Nov 02 14:05:43.226215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:43.226177    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" event={"ID":"fae86fff-9730-4845-b60e-a0eac4259ef8","Type":"ContainerStarted","Data":"ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0"}
Nov 02 14:05:43.226388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:43.226240    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" event={"ID":"fae86fff-9730-4845-b60e-a0eac4259ef8","Type":"ContainerStarted","Data":"9aba97f817f1d3ffa1a2729b38f8acd7087af4e62e91ace7182afa0e1c7bd106"}
Nov 02 14:05:43.275004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:43.274945    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" podStartSLOduration=2.274929172 podStartE2EDuration="2.274929172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:43.273319299 +0000 UTC m=+1334.024623749" watchObservedRunningTime="2026-11-02 14:05:43.274929172 +0000 UTC m=+1334.026233611"
Nov 02 14:05:43.421080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:43.420997    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97adbee5-6ae1-4e63-a452-ae747e102d3c" path="/var/lib/kubelet/pods/97adbee5-6ae1-4e63-a452-ae747e102d3c/volumes"
Nov 02 14:05:43.940996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:43.940962    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:43.947646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:43.947609    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:44.235311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:44.235284    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:44.236148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:44.236126    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:05:44.992635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:44.992594    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"]
Nov 02 14:05:44.992854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:44.992815    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy" containerID="cri-o://09cd10d595a3384caf9ef9b067164087fd7f753f1ee8f87e8881751f8330043c" gracePeriod=30
Nov 02 14:05:47.037943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:47.037878    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"]
Nov 02 14:05:47.047921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:47.047889    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b3b1647-n6f7j"]
Nov 02 14:05:47.397837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:47.397739    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c343511c-58e3-4966-ad2c-bd3a2a9db0bd" path="/var/lib/kubelet/pods/c343511c-58e3-4966-ad2c-bd3a2a9db0bd/volumes"
Nov 02 14:05:48.414265 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:05:48.414223    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:05:50.312412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.312380    5034 generic.go:356] "Generic (PLEG): container finished" podID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerID="09cd10d595a3384caf9ef9b067164087fd7f753f1ee8f87e8881751f8330043c" exitCode=0
Nov 02 14:05:50.312814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.312657    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" event={"ID":"315df3e6-f69e-469f-b78f-ad9bae83a00d","Type":"ContainerDied","Data":"09cd10d595a3384caf9ef9b067164087fd7f753f1ee8f87e8881751f8330043c"}
Nov 02 14:05:50.347994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.347963    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 14:05:50.499262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499221    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-kube-api-access-9h7mt\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-kube-api-access-9h7mt\") pod \"315df3e6-f69e-469f-b78f-ad9bae83a00d\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") "
Nov 02 14:05:50.499439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499332    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-token\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-token\") pod \"315df3e6-f69e-469f-b78f-ad9bae83a00d\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") "
Nov 02 14:05:50.499439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499426    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-data\") pod \"315df3e6-f69e-469f-b78f-ad9bae83a00d\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") "
Nov 02 14:05:50.499579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499487    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-certs\") pod \"315df3e6-f69e-469f-b78f-ad9bae83a00d\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") "
Nov 02 14:05:50.499638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499586    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/315df3e6-f69e-469f-b78f-ad9bae83a00d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315df3e6-f69e-469f-b78f-ad9bae83a00d-istiod-ca-cert\") pod \"315df3e6-f69e-469f-b78f-ad9bae83a00d\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") "
Nov 02 14:05:50.499709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499664    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-socket\") pod \"315df3e6-f69e-469f-b78f-ad9bae83a00d\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") "
Nov 02 14:05:50.499764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499707    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-podinfo\") pod \"315df3e6-f69e-469f-b78f-ad9bae83a00d\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") "
Nov 02 14:05:50.499764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499745    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-envoy\") pod \"315df3e6-f69e-469f-b78f-ad9bae83a00d\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") "
Nov 02 14:05:50.499945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499866    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-credential-socket\") pod \"315df3e6-f69e-469f-b78f-ad9bae83a00d\" (UID: \"315df3e6-f69e-469f-b78f-ad9bae83a00d\") "
Nov 02 14:05:50.499945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.499880    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-data" pod "315df3e6-f69e-469f-b78f-ad9bae83a00d" (UID: "315df3e6-f69e-469f-b78f-ad9bae83a00d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:05:50.500462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.500419    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-certs" pod "315df3e6-f69e-469f-b78f-ad9bae83a00d" (UID: "315df3e6-f69e-469f-b78f-ad9bae83a00d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:05:50.500803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.500609    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-socket" pod "315df3e6-f69e-469f-b78f-ad9bae83a00d" (UID: "315df3e6-f69e-469f-b78f-ad9bae83a00d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:05:50.500998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.500950    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-credential-socket" pod "315df3e6-f69e-469f-b78f-ad9bae83a00d" (UID: "315df3e6-f69e-469f-b78f-ad9bae83a00d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:05:50.502514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.502483    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/315df3e6-f69e-469f-b78f-ad9bae83a00d-istiod-ca-cert" pod "315df3e6-f69e-469f-b78f-ad9bae83a00d" (UID: "315df3e6-f69e-469f-b78f-ad9bae83a00d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:05:50.503349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.503331    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:05:50.503485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.503471    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:05:50.503587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.503575    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/315df3e6-f69e-469f-b78f-ad9bae83a00d-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:05:50.503676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.503666    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:05:50.503761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.503750    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:05:50.504509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.504480    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-token" pod "315df3e6-f69e-469f-b78f-ad9bae83a00d" (UID: "315df3e6-f69e-469f-b78f-ad9bae83a00d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:05:50.505169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.505145    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-kube-api-access-9h7mt" pod "315df3e6-f69e-469f-b78f-ad9bae83a00d" (UID: "315df3e6-f69e-469f-b78f-ad9bae83a00d"). InnerVolumeSpecName "kube-api-access-9h7mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:05:50.505378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.505337    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-envoy" pod "315df3e6-f69e-469f-b78f-ad9bae83a00d" (UID: "315df3e6-f69e-469f-b78f-ad9bae83a00d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:05:50.506771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.506736    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-podinfo" pod "315df3e6-f69e-469f-b78f-ad9bae83a00d" (UID: "315df3e6-f69e-469f-b78f-ad9bae83a00d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:05:50.605328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.605250    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:05:50.605328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.605281    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:05:50.605328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.605290    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h7mt\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-kube-api-access-9h7mt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:05:50.605328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:50.605300    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/315df3e6-f69e-469f-b78f-ad9bae83a00d-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:05:51.333023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.327562    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5" event={"ID":"315df3e6-f69e-469f-b78f-ad9bae83a00d","Type":"ContainerDied","Data":"7e3f6b20823de0e33983e82c335889ba655b51b3f2c144a3fe928b1c9062bc00"}
Nov 02 14:05:51.333023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.327616    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"
Nov 02 14:05:51.333023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.327667    5034 scope.go:122] "RemoveContainer" containerID="09cd10d595a3384caf9ef9b067164087fd7f753f1ee8f87e8881751f8330043c"
Nov 02 14:05:51.423701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.423672    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"]
Nov 02 14:05:51.429529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.429500    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-cjqn5"]
Nov 02 14:05:51.645862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.645762    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"]
Nov 02 14:05:51.646850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.646830    5034 cpu_manager.go:414] "containerMap: removing container" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy"
Nov 02 14:05:51.646918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.646853    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy"
Nov 02 14:05:51.647108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.647086    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" containerName="istio-proxy"
Nov 02 14:05:51.652804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.652766    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.672007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.669749    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-794667-openshift-default-dockercfg-l5w7x\""
Nov 02 14:05:51.676942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.675706    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"]
Nov 02 14:05:51.727726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.727691    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-data\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.728007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.727833    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-podinfo\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.728007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.727920    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfk9\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-kube-api-access-rhfk9\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.728007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.727965    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-envoy\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.728007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.727995    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-certs\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.728269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.728109    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-token\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.728269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.728148    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istiod-ca-cert\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.728269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.728239    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-credential-socket\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.728427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.728345    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-socket\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.838268    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istiod-ca-cert\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.838363    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-credential-socket\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.838441    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-socket\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.838645    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-data\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.838707    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-podinfo\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.838758    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfk9\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-kube-api-access-rhfk9\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.838806    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-envoy\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.838837    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-certs\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.838918    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-token\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.841806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.840473    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istiod-ca-cert\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.920260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.840761    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-credential-socket\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.920393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.865271    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-socket\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.920489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.865505    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-data\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.936808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.887128    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-token\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.936808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.887380    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-certs\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.936808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.887815    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-podinfo\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.936808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.923131    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfk9\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-kube-api-access-rhfk9\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.936808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.936186    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-envoy\") pod \"e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") " pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:51.981808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:51.978419    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:52.326445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:52.326418    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"]
Nov 02 14:05:52.328460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:52.328427    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:05:52.328581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:52.328501    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:05:52.328581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:52.328547    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:05:52.336734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:52.336709    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" event={"ID":"b22ed9db-6d6c-4575-bff2-881a07c7aa04","Type":"ContainerStarted","Data":"77cf48fd0ea2955a17dd2babe243f54bd8105853f336efd0180582d23cb40ffb"}
Nov 02 14:05:53.480808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:53.476354    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="315df3e6-f69e-469f-b78f-ad9bae83a00d" path="/var/lib/kubelet/pods/315df3e6-f69e-469f-b78f-ad9bae83a00d/volumes"
Nov 02 14:05:53.504803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:53.501973    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" event={"ID":"b22ed9db-6d6c-4575-bff2-881a07c7aa04","Type":"ContainerStarted","Data":"eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc"}
Nov 02 14:05:53.541015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:53.540949    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" podStartSLOduration=2.540930426 podStartE2EDuration="2.540930426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:05:53.526281519 +0000 UTC m=+1344.277585971" watchObservedRunningTime="2026-11-02 14:05:53.540930426 +0000 UTC m=+1344.292234876"
Nov 02 14:05:53.986799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:53.984966    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:54.010094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:54.010065    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:54.454866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:54.454744    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:05:54.455619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:05:54.455575    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:06:03.034903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:03.034865    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"]
Nov 02 14:06:03.047628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:03.047603    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f08491-n4t8s"]
Nov 02 14:06:03.399277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:03.399202    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a442fa8-13ed-46ba-a609-97b2d5da5f39" path="/var/lib/kubelet/pods/4a442fa8-13ed-46ba-a609-97b2d5da5f39/volumes"
Nov 02 14:06:05.455733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:05.454939    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"]
Nov 02 14:06:05.455733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:05.455229    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy" containerID="cri-o://269e4cc45cde1310267ab38cbc93d52e1c6e52615c75a107bc7a0fcdcbd81946" gracePeriod=30
Nov 02 14:06:05.492029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:05.490994    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"]
Nov 02 14:06:05.492029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:05.491194    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy" containerID="cri-o://957ffbfa1b3a545540ce027bcf84727d1a4b748f09f3f575ca52c21b8857a36c" gracePeriod=30
Nov 02 14:06:05.494267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:05.494095    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"]
Nov 02 14:06:05.494426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:05.494302    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy" containerID="cri-o://63b9828b5a33e79040d344a8d772fc9c1b062e7e3a2c28514145d16c5948a968" gracePeriod=30
Nov 02 14:06:08.830413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.830374    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"]
Nov 02 14:06:08.848022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.847991    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"]
Nov 02 14:06:08.848183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.848113    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:08.868017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.867988    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f66ca3-openshift-default-dockercfg-z7wqz\""
Nov 02 14:06:08.924857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.924823    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-certs\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:08.925032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.924890    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf07384-0c83-4ffd-94d8-a448e809bf6c-istiod-ca-cert\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:08.925095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.925023    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-token\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:08.925095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.925080    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-data\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:08.925180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.925156    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dljnj\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-kube-api-access-dljnj\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:08.925240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.925225    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-envoy\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:08.925401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.925384    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:08.925473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.925459    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-podinfo\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:08.925535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:08.925506    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-credential-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.027811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.027759    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.028636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.028609    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-podinfo\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.029479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.028495    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.031866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.031839    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-credential-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.031998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.031972    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-certs\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.032114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.032050    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf07384-0c83-4ffd-94d8-a448e809bf6c-istiod-ca-cert\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.032195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.032108    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-token\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.032195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.032159    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-data\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.032303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.032194    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dljnj\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-kube-api-access-dljnj\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.032303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.032282    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-credential-socket\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.032412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.032282    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-envoy\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.032772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.032719    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-data\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.032938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.032914    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf07384-0c83-4ffd-94d8-a448e809bf6c-istiod-ca-cert\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.033349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.033321    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-certs\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.037288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.035346    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-envoy\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.037288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.035827    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-podinfo\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.097887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.088722    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-token\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.097887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.093505    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dljnj\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-kube-api-access-dljnj\") pod \"e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") " pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.201770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.201494    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:09.763880 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:06:09.763844    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf07384_0c83_4ffd_94d8_a448e809bf6c.slice/crio-9c144e4a34d1894ebfb7e66121e1793aee65418c4338fcc1a1538133bee82a00 WatchSource:0}: Error finding container 9c144e4a34d1894ebfb7e66121e1793aee65418c4338fcc1a1538133bee82a00: Status 404 returned error can't find the container with id 9c144e4a34d1894ebfb7e66121e1793aee65418c4338fcc1a1538133bee82a00
Nov 02 14:06:09.765388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.765361    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"]
Nov 02 14:06:09.766687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.766654    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:09.766838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.766717    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:09.766838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:09.766747    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:10.627085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:10.627019    5034 generic.go:356] "Generic (PLEG): container finished" podID="5c53b23c-8143-491f-8704-03fbf789f473" containerID="957ffbfa1b3a545540ce027bcf84727d1a4b748f09f3f575ca52c21b8857a36c" exitCode=0
Nov 02 14:06:10.627738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:10.627705    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" event={"ID":"5c53b23c-8143-491f-8704-03fbf789f473","Type":"ContainerDied","Data":"957ffbfa1b3a545540ce027bcf84727d1a4b748f09f3f575ca52c21b8857a36c"}
Nov 02 14:06:10.631392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:10.631367    5034 generic.go:356] "Generic (PLEG): container finished" podID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerID="63b9828b5a33e79040d344a8d772fc9c1b062e7e3a2c28514145d16c5948a968" exitCode=0
Nov 02 14:06:10.631677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:10.631660    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" event={"ID":"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3","Type":"ContainerDied","Data":"63b9828b5a33e79040d344a8d772fc9c1b062e7e3a2c28514145d16c5948a968"}
Nov 02 14:06:10.634499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:10.634478    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" event={"ID":"abf07384-0c83-4ffd-94d8-a448e809bf6c","Type":"ContainerStarted","Data":"2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4"}
Nov 02 14:06:10.635010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:10.634967    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" event={"ID":"abf07384-0c83-4ffd-94d8-a448e809bf6c","Type":"ContainerStarted","Data":"9c144e4a34d1894ebfb7e66121e1793aee65418c4338fcc1a1538133bee82a00"}
Nov 02 14:06:10.637992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:10.637964    5034 generic.go:356] "Generic (PLEG): container finished" podID="d2851278-2635-49f8-a671-02124f264846" containerID="269e4cc45cde1310267ab38cbc93d52e1c6e52615c75a107bc7a0fcdcbd81946" exitCode=0
Nov 02 14:06:10.638092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:10.638047    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" event={"ID":"d2851278-2635-49f8-a671-02124f264846","Type":"ContainerDied","Data":"269e4cc45cde1310267ab38cbc93d52e1c6e52615c75a107bc7a0fcdcbd81946"}
Nov 02 14:06:10.699432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:10.698471    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" podStartSLOduration=2.698452347 podStartE2EDuration="2.698452347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:10.697362781 +0000 UTC m=+1361.448667220" watchObservedRunningTime="2026-11-02 14:06:10.698452347 +0000 UTC m=+1361.449756788"
Nov 02 14:06:11.203956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.203861    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:11.210810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.210743    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:11.322499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.317957    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.426167    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.426702    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istiod-ca-cert\") pod \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") "
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.426790    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-envoy\") pod \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") "
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.426829    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-kube-api-access-n5n5g\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-kube-api-access-n5n5g\") pod \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") "
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.426879    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-token\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-token\") pod \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") "
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.426909    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-certs\") pod \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") "
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.426960    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-socket\") pod \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") "
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.427056    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-data\") pod \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") "
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.427088    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-credential-socket\") pod \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") "
Nov 02 14:06:11.428798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.427143    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-podinfo\") pod \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\" (UID: \"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3\") "
Nov 02 14:06:11.434819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.430156    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-podinfo" pod "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" (UID: "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:06:11.434819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.430446    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-certs" pod "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" (UID: "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.434819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.430625    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-socket" pod "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" (UID: "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.434819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.430743    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-data" pod "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" (UID: "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.434819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.430894    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-envoy" pod "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" (UID: "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.434819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.430941    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-credential-socket" pod "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" (UID: "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.434819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.431233    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istiod-ca-cert" pod "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" (UID: "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:11.434819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.432610    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-token" pod "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" (UID: "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:11.436233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.435246    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-kube-api-access-n5n5g" pod "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" (UID: "f0ce2dc0-9f5e-47e8-970b-d407bb0036e3"). InnerVolumeSpecName "kube-api-access-n5n5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:11.462894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.462870    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:06:11.528675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.528641    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-certs\") pod \"5c53b23c-8143-491f-8704-03fbf789f473\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") "
Nov 02 14:06:11.528872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.528693    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-credential-socket\") pod \"5c53b23c-8143-491f-8704-03fbf789f473\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") "
Nov 02 14:06:11.528872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.528793    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2851278-2635-49f8-a671-02124f264846-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2851278-2635-49f8-a671-02124f264846-istiod-ca-cert\") pod \"d2851278-2635-49f8-a671-02124f264846\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") "
Nov 02 14:06:11.529003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.528885    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-envoy\") pod \"d2851278-2635-49f8-a671-02124f264846\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") "
Nov 02 14:06:11.529003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.528939    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-kube-api-access-728xr\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-kube-api-access-728xr\") pod \"d2851278-2635-49f8-a671-02124f264846\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") "
Nov 02 14:06:11.529003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.528972    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2851278-2635-49f8-a671-02124f264846-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2851278-2635-49f8-a671-02124f264846-istio-podinfo\") pod \"d2851278-2635-49f8-a671-02124f264846\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") "
Nov 02 14:06:11.529164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529010    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-kube-api-access-kg68n\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-kube-api-access-kg68n\") pod \"5c53b23c-8143-491f-8704-03fbf789f473\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") "
Nov 02 14:06:11.529164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529051    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-data\") pod \"5c53b23c-8143-491f-8704-03fbf789f473\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") "
Nov 02 14:06:11.529164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529085    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c53b23c-8143-491f-8704-03fbf789f473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c53b23c-8143-491f-8704-03fbf789f473-istio-podinfo\") pod \"5c53b23c-8143-491f-8704-03fbf789f473\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") "
Nov 02 14:06:11.529164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529150    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-certs\") pod \"d2851278-2635-49f8-a671-02124f264846\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") "
Nov 02 14:06:11.529374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529206    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-data\") pod \"d2851278-2635-49f8-a671-02124f264846\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") "
Nov 02 14:06:11.529374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529268    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-credential-socket\") pod \"d2851278-2635-49f8-a671-02124f264846\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") "
Nov 02 14:06:11.529374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529336    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-istio-token\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-istio-token\") pod \"5c53b23c-8143-491f-8704-03fbf789f473\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") "
Nov 02 14:06:11.529531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529392    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-istio-token\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-istio-token\") pod \"d2851278-2635-49f8-a671-02124f264846\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") "
Nov 02 14:06:11.529531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529425    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-socket\") pod \"d2851278-2635-49f8-a671-02124f264846\" (UID: \"d2851278-2635-49f8-a671-02124f264846\") "
Nov 02 14:06:11.529531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529457    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c53b23c-8143-491f-8704-03fbf789f473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c53b23c-8143-491f-8704-03fbf789f473-istiod-ca-cert\") pod \"5c53b23c-8143-491f-8704-03fbf789f473\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") "
Nov 02 14:06:11.529735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529535    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-envoy\") pod \"5c53b23c-8143-491f-8704-03fbf789f473\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") "
Nov 02 14:06:11.529735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.529576    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-socket\") pod \"5c53b23c-8143-491f-8704-03fbf789f473\" (UID: \"5c53b23c-8143-491f-8704-03fbf789f473\") "
Nov 02 14:06:11.530771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.530606    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.530771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.530637    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.530771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.530653    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.530771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.530668    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.530771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.530681    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.530771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.530695    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5n5g\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-kube-api-access-n5n5g\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.530771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.530709    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.530771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.530723    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.530771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.530735    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.531418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.531347    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-certs" pod "5c53b23c-8143-491f-8704-03fbf789f473" (UID: "5c53b23c-8143-491f-8704-03fbf789f473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.531788 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.531575    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-credential-socket" pod "5c53b23c-8143-491f-8704-03fbf789f473" (UID: "5c53b23c-8143-491f-8704-03fbf789f473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.531927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.531904    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2851278-2635-49f8-a671-02124f264846-istiod-ca-cert" pod "d2851278-2635-49f8-a671-02124f264846" (UID: "d2851278-2635-49f8-a671-02124f264846"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:11.534977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.534079    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-data" pod "d2851278-2635-49f8-a671-02124f264846" (UID: "d2851278-2635-49f8-a671-02124f264846"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.534977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.534176    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c53b23c-8143-491f-8704-03fbf789f473-istiod-ca-cert" pod "5c53b23c-8143-491f-8704-03fbf789f473" (UID: "5c53b23c-8143-491f-8704-03fbf789f473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:11.534977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.534370    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-socket" pod "d2851278-2635-49f8-a671-02124f264846" (UID: "d2851278-2635-49f8-a671-02124f264846"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.534977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.534468    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-socket" pod "5c53b23c-8143-491f-8704-03fbf789f473" (UID: "5c53b23c-8143-491f-8704-03fbf789f473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.534977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.534495    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-envoy" pod "d2851278-2635-49f8-a671-02124f264846" (UID: "d2851278-2635-49f8-a671-02124f264846"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.534977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.534678    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-credential-socket" pod "d2851278-2635-49f8-a671-02124f264846" (UID: "d2851278-2635-49f8-a671-02124f264846"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.535462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.535064    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-certs" pod "d2851278-2635-49f8-a671-02124f264846" (UID: "d2851278-2635-49f8-a671-02124f264846"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.535877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.535855    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-data" pod "5c53b23c-8143-491f-8704-03fbf789f473" (UID: "5c53b23c-8143-491f-8704-03fbf789f473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.537963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.537936    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-kube-api-access-kg68n" pod "5c53b23c-8143-491f-8704-03fbf789f473" (UID: "5c53b23c-8143-491f-8704-03fbf789f473"). InnerVolumeSpecName "kube-api-access-kg68n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:11.538061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.537980    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c53b23c-8143-491f-8704-03fbf789f473-istio-podinfo" pod "5c53b23c-8143-491f-8704-03fbf789f473" (UID: "5c53b23c-8143-491f-8704-03fbf789f473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:06:11.539107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.539080    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-istio-token" pod "5c53b23c-8143-491f-8704-03fbf789f473" (UID: "5c53b23c-8143-491f-8704-03fbf789f473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:11.539188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.539172    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-istio-token" pod "d2851278-2635-49f8-a671-02124f264846" (UID: "d2851278-2635-49f8-a671-02124f264846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:11.539319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.539297    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2851278-2635-49f8-a671-02124f264846-istio-podinfo" pod "d2851278-2635-49f8-a671-02124f264846" (UID: "d2851278-2635-49f8-a671-02124f264846"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:06:11.539450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.539432    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-kube-api-access-728xr" pod "d2851278-2635-49f8-a671-02124f264846" (UID: "d2851278-2635-49f8-a671-02124f264846"). InnerVolumeSpecName "kube-api-access-728xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:11.541148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.541121    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-envoy" pod "5c53b23c-8143-491f-8704-03fbf789f473" (UID: "5c53b23c-8143-491f-8704-03fbf789f473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:11.633439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.633359    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.633952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.633924    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.633949    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.633978    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.633988    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.633999    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634007    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c53b23c-8143-491f-8704-03fbf789f473-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634015    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634024    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634031    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634055    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634064    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2851278-2635-49f8-a671-02124f264846-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634072    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2851278-2635-49f8-a671-02124f264846-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634080    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-728xr\" (UniqueName: \"kubernetes.io/projected/d2851278-2635-49f8-a671-02124f264846-kube-api-access-728xr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634089    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2851278-2635-49f8-a671-02124f264846-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634100    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg68n\" (UniqueName: \"kubernetes.io/projected/5c53b23c-8143-491f-8704-03fbf789f473-kube-api-access-kg68n\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634109    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c53b23c-8143-491f-8704-03fbf789f473-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.634395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.634132    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c53b23c-8143-491f-8704-03fbf789f473-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:11.647857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.647829    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"
Nov 02 14:06:11.647857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.647846    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc" event={"ID":"d2851278-2635-49f8-a671-02124f264846","Type":"ContainerDied","Data":"d825725940b4e21b2d62579a84e7ed783c4ad57daca951a2eff3bd66ed376c9f"}
Nov 02 14:06:11.648029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.647919    5034 scope.go:122] "RemoveContainer" containerID="269e4cc45cde1310267ab38cbc93d52e1c6e52615c75a107bc7a0fcdcbd81946"
Nov 02 14:06:11.649442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.649422    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf" event={"ID":"5c53b23c-8143-491f-8704-03fbf789f473","Type":"ContainerDied","Data":"aa6616b9441e6aaa3b548c2c842f70d8e21f63d7120941c6e81391bdb16bed7d"}
Nov 02 14:06:11.649548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.649520    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"
Nov 02 14:06:11.652288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.652267    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"
Nov 02 14:06:11.652399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.652311    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk" event={"ID":"f0ce2dc0-9f5e-47e8-970b-d407bb0036e3","Type":"ContainerDied","Data":"d10f9a062b3bf439d95e28848813cae8362eb57f75b32fdf569351e1463aaa73"}
Nov 02 14:06:11.652719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.652702    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:11.653673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.653653    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:06:11.665571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.665555    5034 scope.go:122] "RemoveContainer" containerID="957ffbfa1b3a545540ce027bcf84727d1a4b748f09f3f575ca52c21b8857a36c"
Nov 02 14:06:11.710134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.695013    5034 scope.go:122] "RemoveContainer" containerID="63b9828b5a33e79040d344a8d772fc9c1b062e7e3a2c28514145d16c5948a968"
Nov 02 14:06:11.713382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.713361    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"]
Nov 02 14:06:11.717357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.717334    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-stgkc"]
Nov 02 14:06:11.752701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.752667    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"]
Nov 02 14:06:11.756236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.756217    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-z8cgf"]
Nov 02 14:06:11.902672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.902586    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"]
Nov 02 14:06:11.906345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:11.906313    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-zfqdk"]
Nov 02 14:06:13.406171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:13.406134    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c53b23c-8143-491f-8704-03fbf789f473" path="/var/lib/kubelet/pods/5c53b23c-8143-491f-8704-03fbf789f473/volumes"
Nov 02 14:06:13.406794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:13.406766    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2851278-2635-49f8-a671-02124f264846" path="/var/lib/kubelet/pods/d2851278-2635-49f8-a671-02124f264846/volumes"
Nov 02 14:06:13.407414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:13.407398    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" path="/var/lib/kubelet/pods/f0ce2dc0-9f5e-47e8-970b-d407bb0036e3/volumes"
Nov 02 14:06:15.030925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.030896    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"]
Nov 02 14:06:15.031649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.031628    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy"
Nov 02 14:06:15.031649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.031646    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy"
Nov 02 14:06:15.031885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.031669    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy"
Nov 02 14:06:15.031885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.031674    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy"
Nov 02 14:06:15.031885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.031684    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy"
Nov 02 14:06:15.031885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.031689    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy"
Nov 02 14:06:15.031885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.031839    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2851278-2635-49f8-a671-02124f264846" containerName="istio-proxy"
Nov 02 14:06:15.031885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.031852    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c53b23c-8143-491f-8704-03fbf789f473" containerName="istio-proxy"
Nov 02 14:06:15.031885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.031859    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0ce2dc0-9f5e-47e8-970b-d407bb0036e3" containerName="istio-proxy"
Nov 02 14:06:15.037609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.037589    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:15.056981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.056958    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f66ca3\""
Nov 02 14:06:15.060219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.060197    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"]
Nov 02 14:06:15.069126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.069103    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f66ca3-dockercfg-wcsw5\""
Nov 02 14:06:15.074939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.074918    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34b1998f-clhxm"]
Nov 02 14:06:15.107473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.107440    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxhgq\" (UniqueName: \"kubernetes.io/projected/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-kube-api-access-nxhgq\") pod \"payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5\" (UID: \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:15.107913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.107888    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5\" (UID: \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:15.215801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.211752    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxhgq\" (UniqueName: \"kubernetes.io/projected/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-kube-api-access-nxhgq\") pod \"payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5\" (UID: \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:15.215801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.212034    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5\" (UID: \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:15.215801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.213126    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5\" (UID: \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:15.293884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.293683    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"]
Nov 02 14:06:15.303024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.302986    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"]
Nov 02 14:06:15.309516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.308518    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxhgq\" (UniqueName: \"kubernetes.io/projected/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-kube-api-access-nxhgq\") pod \"payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5\" (UID: \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\") " pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:15.326121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.326091    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:15.336914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.336877    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"]
Nov 02 14:06:15.353000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.352975    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:15.434838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.431867    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858\" (UID: \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:15.434838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.432073    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jh6r\" (UniqueName: \"kubernetes.io/projected/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-kube-api-access-8jh6r\") pod \"payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858\" (UID: \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:15.437808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.437347    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cba9546-58bd-4185-9c00-1d0043b08f41" path="/var/lib/kubelet/pods/2cba9546-58bd-4185-9c00-1d0043b08f41/volumes"
Nov 02 14:06:15.534502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.533964    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jh6r\" (UniqueName: \"kubernetes.io/projected/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-kube-api-access-8jh6r\") pod \"payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858\" (UID: \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:15.534502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.534311    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858\" (UID: \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:15.535190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.535160    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858\" (UID: \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:15.561336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.561224    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jh6r\" (UniqueName: \"kubernetes.io/projected/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-kube-api-access-8jh6r\") pod \"payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858\" (UID: \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:15.646083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.646049    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:15.751472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:15.751385    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"]
Nov 02 14:06:15.753335 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:06:15.753304    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb5e6e8_b98a_437f_8799_e47bf23e98aa.slice/crio-12e8833efc88d43b14d22058676343ab4282a092f1899b79b4949efb68b1975f WatchSource:0}: Error finding container 12e8833efc88d43b14d22058676343ab4282a092f1899b79b4949efb68b1975f: Status 404 returned error can't find the container with id 12e8833efc88d43b14d22058676343ab4282a092f1899b79b4949efb68b1975f
Nov 02 14:06:16.050099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.049895    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"]
Nov 02 14:06:16.052482 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:06:16.052415    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c24e50_85aa_4321_b00b_fa9de3b3aa55.slice/crio-98e349dc361f6ac877d778692609055a7e6f3f6e3cceeabb73ee903d15e372df WatchSource:0}: Error finding container 98e349dc361f6ac877d778692609055a7e6f3f6e3cceeabb73ee903d15e372df: Status 404 returned error can't find the container with id 98e349dc361f6ac877d778692609055a7e6f3f6e3cceeabb73ee903d15e372df
Nov 02 14:06:16.089273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.087915    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:06:16.089273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.087973    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:06:16.749675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.749638    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5" event={"ID":"ffb5e6e8-b98a-437f-8799-e47bf23e98aa","Type":"ContainerStarted","Data":"cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196"}
Nov 02 14:06:16.749892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.749693    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5" event={"ID":"ffb5e6e8-b98a-437f-8799-e47bf23e98aa","Type":"ContainerStarted","Data":"12e8833efc88d43b14d22058676343ab4282a092f1899b79b4949efb68b1975f"}
Nov 02 14:06:16.749892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.749734    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:16.751467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.751440    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858" event={"ID":"b9c24e50-85aa-4321-b00b-fa9de3b3aa55","Type":"ContainerStarted","Data":"74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584"}
Nov 02 14:06:16.751590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.751481    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858" event={"ID":"b9c24e50-85aa-4321-b00b-fa9de3b3aa55","Type":"ContainerStarted","Data":"98e349dc361f6ac877d778692609055a7e6f3f6e3cceeabb73ee903d15e372df"}
Nov 02 14:06:16.751590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.751524    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:16.859880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.859830    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5" podStartSLOduration=1.859815684 podStartE2EDuration="1.859815684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:16.811510461 +0000 UTC m=+1367.562814899" watchObservedRunningTime="2026-11-02 14:06:16.859815684 +0000 UTC m=+1367.611120116"
Nov 02 14:06:16.860222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:16.860197    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858" podStartSLOduration=1.86019005 podStartE2EDuration="1.86019005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:16.855564554 +0000 UTC m=+1367.606868993" watchObservedRunningTime="2026-11-02 14:06:16.86019005 +0000 UTC m=+1367.611494487"
Nov 02 14:06:18.426684 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:06:18.426633    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:06:18.768256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:18.768224    5034 generic.go:356] "Generic (PLEG): container finished" podID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerID="a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de" exitCode=0
Nov 02 14:06:18.768420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:18.768296    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" event={"ID":"9109731a-74f9-4294-92f3-dc6da573a3c8","Type":"ContainerDied","Data":"a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de"}
Nov 02 14:06:18.768733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:18.768720    5034 scope.go:122] "RemoveContainer" containerID="a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de"
Nov 02 14:06:18.770526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:18.770492    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:18.770638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:18.770582    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:18.770638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:18.770624    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:19.796804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:19.796078    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" event={"ID":"9109731a-74f9-4294-92f3-dc6da573a3c8","Type":"ContainerStarted","Data":"144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f"}
Nov 02 14:06:20.639998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:20.639963    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:06:20.640181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:20.640117    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 14:06:20.640181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:20.640163    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 14:06:21.640197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:21.640166    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 14:06:21.640647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:21.640220    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 14:06:22.639967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:22.639633    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 14:06:22.639967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:22.639690    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 14:06:23.639792 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.639295    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 14:06:23.639792 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.639354    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 14:06:23.748954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.748923    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"]
Nov 02 14:06:23.756182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.756095    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:23.905856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.873257    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b7489f-openshift-default-dockercfg-x947w\""
Nov 02 14:06:23.979804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.977798    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c57591-6c30-490a-b348-e97fbbd80319-istiod-ca-cert\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:23.979804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.977872    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-credential-socket\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:23.979804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.977937    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-socket\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:23.979804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.977973    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c57591-6c30-490a-b348-e97fbbd80319-istio-podinfo\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:23.979804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.978039    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4bj2\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-kube-api-access-s4bj2\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:23.979804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.978192    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-data\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:23.979804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.978274    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-istio-token\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:23.979804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.978319    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-certs\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:23.979804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:23.978347    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-envoy\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.251843    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"]
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.252747    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4bj2\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-kube-api-access-s4bj2\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.252929    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-data\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.253004    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-istio-token\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.253040    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-certs\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.253065    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-envoy\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.253185    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c57591-6c30-490a-b348-e97fbbd80319-istiod-ca-cert\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.253229    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-credential-socket\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.253284    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-socket\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.253317    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c57591-6c30-490a-b348-e97fbbd80319-istio-podinfo\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.253429    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-data\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.254273    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-credential-socket\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.254891    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c57591-6c30-490a-b348-e97fbbd80319-istiod-ca-cert\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.255132    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-certs\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.255657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.255346    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-socket\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.256889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.256303    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-envoy\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.257015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.256985    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c57591-6c30-490a-b348-e97fbbd80319-istio-podinfo\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.344449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.343937    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4bj2\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-kube-api-access-s4bj2\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.345684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.345654    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-istio-token\") pod \"e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") " pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.524851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.523079    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:24.640961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.640505    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Nov 02 14:06:24.640961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.640591    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Nov 02 14:06:24.946552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.946518    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"]
Nov 02 14:06:24.947328 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:06:24.947305    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c57591_6c30_490a_b348_e97fbbd80319.slice/crio-1ce16cabe02289568b1e627e9cdd027df19188b1ec389a49452d1f2e2394204b WatchSource:0}: Error finding container 1ce16cabe02289568b1e627e9cdd027df19188b1ec389a49452d1f2e2394204b: Status 404 returned error can't find the container with id 1ce16cabe02289568b1e627e9cdd027df19188b1ec389a49452d1f2e2394204b
Nov 02 14:06:24.949899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.949865    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:24.950004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.949942    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:24.950004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:24.949985    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:25.268802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:25.265881    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" event={"ID":"15c57591-6c30-490a-b348-e97fbbd80319","Type":"ContainerStarted","Data":"dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc"}
Nov 02 14:06:25.268802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:25.265947    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" event={"ID":"15c57591-6c30-490a-b348-e97fbbd80319","Type":"ContainerStarted","Data":"1ce16cabe02289568b1e627e9cdd027df19188b1ec389a49452d1f2e2394204b"}
Nov 02 14:06:25.326978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:25.326147    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" podStartSLOduration=2.32612693 podStartE2EDuration="2.32612693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:25.320692917 +0000 UTC m=+1376.071997358" watchObservedRunningTime="2026-11-02 14:06:25.32612693 +0000 UTC m=+1376.077431372"
Nov 02 14:06:25.528251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:25.524922    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:25.793983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:25.792498    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:06:25.793983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:25.793757    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:06:25.803038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:25.802978    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:06:26.529657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:26.529624    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:27.318801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:27.317892    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:27.319296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:27.319233    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:06:27.762154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:27.762130    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:06:27.762326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:27.762269    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:06:29.178055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:29.178021    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"]
Nov 02 14:06:29.178485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:29.178263    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy" containerID="cri-o://25eb5e59090bd31eb2022f1ef97a6c6b1409b81f338cc52cdb89e293cefb184f" gracePeriod=30
Nov 02 14:06:29.637133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:29.637100    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"]
Nov 02 14:06:29.640469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:29.640448    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:29.665369 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:29.665334    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b7489f-dockercfg-xxspv\""
Nov 02 14:06:29.675802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:29.669618    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b7489f\""
Nov 02 14:06:29.873930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:29.870580    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"]
Nov 02 14:06:30.000864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.000609    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394d2a81-57f2-4c32-989f-492a4595968c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f\" (UID: \"394d2a81-57f2-4c32-989f-492a4595968c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:30.001063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.000931    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp29f\" (UniqueName: \"kubernetes.io/projected/394d2a81-57f2-4c32-989f-492a4595968c-kube-api-access-xp29f\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f\" (UID: \"394d2a81-57f2-4c32-989f-492a4595968c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:30.102506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.102472    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394d2a81-57f2-4c32-989f-492a4595968c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f\" (UID: \"394d2a81-57f2-4c32-989f-492a4595968c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:30.102712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.102661    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp29f\" (UniqueName: \"kubernetes.io/projected/394d2a81-57f2-4c32-989f-492a4595968c-kube-api-access-xp29f\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f\" (UID: \"394d2a81-57f2-4c32-989f-492a4595968c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:30.117981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.117949    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b7489f\""
Nov 02 14:06:30.123585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.123553    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394d2a81-57f2-4c32-989f-492a4595968c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f\" (UID: \"394d2a81-57f2-4c32-989f-492a4595968c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:30.128024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.127970    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp29f\" (UniqueName: \"kubernetes.io/projected/394d2a81-57f2-4c32-989f-492a4595968c-kube-api-access-xp29f\") pod \"payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f\" (UID: \"394d2a81-57f2-4c32-989f-492a4595968c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:30.257286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.257212    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:30.275947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.275918    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:06:30.276251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.276215    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:06:30.575972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:30.575930    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"]
Nov 02 14:06:31.372291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:31.372248    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f" event={"ID":"394d2a81-57f2-4c32-989f-492a4595968c","Type":"ContainerStarted","Data":"44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba"}
Nov 02 14:06:31.372904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:31.372876    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f" event={"ID":"394d2a81-57f2-4c32-989f-492a4595968c","Type":"ContainerStarted","Data":"f6e49d680b77ed2b2d7558514b94044191247063b0c1b224f3d07203df3492e1"}
Nov 02 14:06:31.374060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:31.374030    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:31.437630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:31.437567    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f" podStartSLOduration=2.437551964 podStartE2EDuration="2.437551964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:31.429559566 +0000 UTC m=+1382.180864005" watchObservedRunningTime="2026-11-02 14:06:31.437551964 +0000 UTC m=+1382.188856403"
Nov 02 14:06:33.390332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:33.390262    5034 generic.go:356] "Generic (PLEG): container finished" podID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerID="ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0" exitCode=0
Nov 02 14:06:33.394939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:33.394911    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" event={"ID":"fae86fff-9730-4845-b60e-a0eac4259ef8","Type":"ContainerDied","Data":"ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0"}
Nov 02 14:06:33.395411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:33.395221    5034 scope.go:122] "RemoveContainer" containerID="ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0"
Nov 02 14:06:33.396758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:33.396729    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:33.396877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:33.396826    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:33.396923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:33.396886    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:34.409808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.409750    5034 generic.go:356] "Generic (PLEG): container finished" podID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerID="25eb5e59090bd31eb2022f1ef97a6c6b1409b81f338cc52cdb89e293cefb184f" exitCode=0
Nov 02 14:06:34.410348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.409808    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" event={"ID":"4226b2d7-e1e2-4e5f-bcc6-7401535821b5","Type":"ContainerDied","Data":"25eb5e59090bd31eb2022f1ef97a6c6b1409b81f338cc52cdb89e293cefb184f"}
Nov 02 14:06:34.413484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.413462    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" event={"ID":"fae86fff-9730-4845-b60e-a0eac4259ef8","Type":"ContainerStarted","Data":"368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7"}
Nov 02 14:06:34.560722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.560693    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 14:06:34.717747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.717707    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-token\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-token\") pod \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") "
Nov 02 14:06:34.717986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.717802    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-data\") pod \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") "
Nov 02 14:06:34.717986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.717884    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-credential-socket\") pod \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") "
Nov 02 14:06:34.717986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.717902    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-envoy\") pod \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") "
Nov 02 14:06:34.717986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.717946    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-kube-api-access-z7wtt\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-kube-api-access-z7wtt\") pod \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") "
Nov 02 14:06:34.717986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.717968    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-podinfo\") pod \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") "
Nov 02 14:06:34.718278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.718066    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-socket\") pod \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") "
Nov 02 14:06:34.718278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.718138    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-certs\") pod \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") "
Nov 02 14:06:34.718278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.718175    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istiod-ca-cert\") pod \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\" (UID: \"4226b2d7-e1e2-4e5f-bcc6-7401535821b5\") "
Nov 02 14:06:34.718916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.718892    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istiod-ca-cert" pod "4226b2d7-e1e2-4e5f-bcc6-7401535821b5" (UID: "4226b2d7-e1e2-4e5f-bcc6-7401535821b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:34.719548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.719519    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-socket" pod "4226b2d7-e1e2-4e5f-bcc6-7401535821b5" (UID: "4226b2d7-e1e2-4e5f-bcc6-7401535821b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.719683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.719639    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-credential-socket" pod "4226b2d7-e1e2-4e5f-bcc6-7401535821b5" (UID: "4226b2d7-e1e2-4e5f-bcc6-7401535821b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.719683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.719662    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-data" pod "4226b2d7-e1e2-4e5f-bcc6-7401535821b5" (UID: "4226b2d7-e1e2-4e5f-bcc6-7401535821b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.719845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.719719    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-certs" pod "4226b2d7-e1e2-4e5f-bcc6-7401535821b5" (UID: "4226b2d7-e1e2-4e5f-bcc6-7401535821b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.721945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.721900    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-podinfo" pod "4226b2d7-e1e2-4e5f-bcc6-7401535821b5" (UID: "4226b2d7-e1e2-4e5f-bcc6-7401535821b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:06:34.721945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.721910    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-token" pod "4226b2d7-e1e2-4e5f-bcc6-7401535821b5" (UID: "4226b2d7-e1e2-4e5f-bcc6-7401535821b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:34.722297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.722047    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-kube-api-access-z7wtt" pod "4226b2d7-e1e2-4e5f-bcc6-7401535821b5" (UID: "4226b2d7-e1e2-4e5f-bcc6-7401535821b5"). InnerVolumeSpecName "kube-api-access-z7wtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:34.724913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.724889    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-envoy" pod "4226b2d7-e1e2-4e5f-bcc6-7401535821b5" (UID: "4226b2d7-e1e2-4e5f-bcc6-7401535821b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:34.820388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.820295    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.820388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.820329    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.820388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.820342    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.820388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.820353    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.820388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.820365    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.820388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.820376    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.820388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.820389    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.820753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.820402    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7wtt\" (UniqueName: \"kubernetes.io/projected/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-kube-api-access-z7wtt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.820753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.820414    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4226b2d7-e1e2-4e5f-bcc6-7401535821b5-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:34.940963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.940930    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:06:34.941134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.941093    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 14:06:34.941193 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.941138    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 14:06:34.953387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.953361    5034 scope.go:122] "RemoveContainer" containerID="25eb5e59090bd31eb2022f1ef97a6c6b1409b81f338cc52cdb89e293cefb184f"
Nov 02 14:06:34.965753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:34.965734    5034 scope.go:122] "RemoveContainer" containerID="edc64545a49cb34cbb8dd1b76a415cf670bceceede0113eb4a249990351cda88"
Nov 02 14:06:35.008620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.008593    5034 scope.go:122] "RemoveContainer" containerID="4f597b63f04b5f0b7746b99009e514727428f33065034aa7cec67b093e56fdcb"
Nov 02 14:06:35.018230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.018208    5034 scope.go:122] "RemoveContainer" containerID="6cb96cf3bf5e9ca3997750ce3665e8b1114930e674cbc83f2aea8e49feb90f59"
Nov 02 14:06:35.027256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.027237    5034 scope.go:122] "RemoveContainer" containerID="88b337407b57d3bfba6f4a1069973c9cd4dbf10fc255a5eac7cbff23bb4fb30d"
Nov 02 14:06:35.036203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.036185    5034 scope.go:122] "RemoveContainer" containerID="dc5c6766cfd331d624ebf15f615db171034ab4bb2c1cc535f29e1ca07932967a"
Nov 02 14:06:35.044960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.044932    5034 scope.go:122] "RemoveContainer" containerID="1a713596fd2e2cc9a699e6a5521ec23d34a348d6040e87ab7a2705339f08854d"
Nov 02 14:06:35.053448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.053427    5034 scope.go:122] "RemoveContainer" containerID="de67e7d63ea223720a55597bbce41f628a181cf5acbd99938e3ed0a32b806844"
Nov 02 14:06:35.062435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.062416    5034 scope.go:122] "RemoveContainer" containerID="fad2e8ee50b96f9ff0e5743a666c7834514a4a390b38e98b1e297e7c4c500cff"
Nov 02 14:06:35.421047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.421021    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"
Nov 02 14:06:35.421575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.421541    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk" event={"ID":"4226b2d7-e1e2-4e5f-bcc6-7401535821b5","Type":"ContainerDied","Data":"9ad11540e0e8cfd65153f3421f17e49d0903c08bd6217eb49dd7fcda32193670"}
Nov 02 14:06:35.530361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.530327    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"]
Nov 02 14:06:35.534830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.534803    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-l7fdk"]
Nov 02 14:06:35.953808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.951089    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 14:06:35.953808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:35.951151    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 14:06:36.941068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:36.941031    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 14:06:36.941460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:36.941085    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 14:06:37.405736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:37.405702    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" path="/var/lib/kubelet/pods/4226b2d7-e1e2-4e5f-bcc6-7401535821b5/volumes"
Nov 02 14:06:37.941772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:37.941735    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 14:06:37.942201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:37.941814    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 14:06:38.942007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:38.941973    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body=
Nov 02 14:06:38.942434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:38.942021    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused"
Nov 02 14:06:39.945046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:39.945018    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:06:39.945510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:39.945497    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:06:39.946149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:39.946131    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:06:43.395638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:43.395552    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:06:43.905403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:43.905363    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"]
Nov 02 14:06:43.906798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:43.906755    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy"
Nov 02 14:06:43.906949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:43.906935    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy"
Nov 02 14:06:43.907267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:43.907250    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4226b2d7-e1e2-4e5f-bcc6-7401535821b5" containerName="istio-proxy"
Nov 02 14:06:43.948996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:43.939235    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"]
Nov 02 14:06:43.948996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:43.939352    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:43.961454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:43.961425    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fb206f-dockercfg-hztpk\""
Nov 02 14:06:43.975436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:43.975409    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fb206f\""
Nov 02 14:06:44.077955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.077919    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074235ec-9cde-4e11-b5bf-0160a6c5729a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr\" (UID: \"074235ec-9cde-4e11-b5bf-0160a6c5729a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:44.078132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.078104    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpctn\" (UniqueName: \"kubernetes.io/projected/074235ec-9cde-4e11-b5bf-0160a6c5729a-kube-api-access-lpctn\") pod \"payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr\" (UID: \"074235ec-9cde-4e11-b5bf-0160a6c5729a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:44.179380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.179295    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpctn\" (UniqueName: \"kubernetes.io/projected/074235ec-9cde-4e11-b5bf-0160a6c5729a-kube-api-access-lpctn\") pod \"payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr\" (UID: \"074235ec-9cde-4e11-b5bf-0160a6c5729a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:44.179548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.179500    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074235ec-9cde-4e11-b5bf-0160a6c5729a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr\" (UID: \"074235ec-9cde-4e11-b5bf-0160a6c5729a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:44.180215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.180194    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074235ec-9cde-4e11-b5bf-0160a6c5729a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr\" (UID: \"074235ec-9cde-4e11-b5bf-0160a6c5729a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:44.205903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.205874    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpctn\" (UniqueName: \"kubernetes.io/projected/074235ec-9cde-4e11-b5bf-0160a6c5729a-kube-api-access-lpctn\") pod \"payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr\" (UID: \"074235ec-9cde-4e11-b5bf-0160a6c5729a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:44.266047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.266024    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:44.502208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.502164    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"]
Nov 02 14:06:44.502698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.502414    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" containerID="cri-o://337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612" gracePeriod=30
Nov 02 14:06:44.507846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.507817    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"]
Nov 02 14:06:44.508026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.508006    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy" containerID="cri-o://db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df" gracePeriod=30
Nov 02 14:06:44.585957 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:06:44.585120    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074235ec_9cde_4e11_b5bf_0160a6c5729a.slice/crio-321123769b4b70fcff44cb5140f51073ab43c6090e08ba754185f3ed5ef3fe9e WatchSource:0}: Error finding container 321123769b4b70fcff44cb5140f51073ab43c6090e08ba754185f3ed5ef3fe9e: Status 404 returned error can't find the container with id 321123769b4b70fcff44cb5140f51073ab43c6090e08ba754185f3ed5ef3fe9e
Nov 02 14:06:44.585957 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:44.585591    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"]
Nov 02 14:06:45.117449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:45.117409    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:06:45.117647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:45.117481    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:06:45.609362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:45.609320    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr" event={"ID":"074235ec-9cde-4e11-b5bf-0160a6c5729a","Type":"ContainerStarted","Data":"321123769b4b70fcff44cb5140f51073ab43c6090e08ba754185f3ed5ef3fe9e"}
Nov 02 14:06:46.621058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:46.621010    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr" event={"ID":"074235ec-9cde-4e11-b5bf-0160a6c5729a","Type":"ContainerStarted","Data":"258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3"}
Nov 02 14:06:46.621548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:46.621095    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:46.667435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:46.667372    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr" podStartSLOduration=1.795507464 podStartE2EDuration="3.667354041s" totalImagesPullingTime="1.871846577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:06:46.665576139 +0000 UTC m=+1397.416880576" watchObservedRunningTime="2026-11-02 14:06:46.667354041 +0000 UTC m=+1397.418658487"
Nov 02 14:06:48.426322 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:06:48.426280    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:06:50.119515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.119482    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 14:06:50.149905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.149877    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 14:06:50.231027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.230981    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istiod-ca-cert\") pod \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") "
Nov 02 14:06:50.231222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231056    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-envoy\") pod \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") "
Nov 02 14:06:50.231222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231129    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-token\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-token\") pod \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") "
Nov 02 14:06:50.231350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231250    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-credential-socket\") pod \"58b78b2a-6676-4572-9b89-7b8e60df80f6\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") "
Nov 02 14:06:50.231350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231293    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-data\") pod \"58b78b2a-6676-4572-9b89-7b8e60df80f6\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") "
Nov 02 14:06:50.231464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231353    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-socket\") pod \"58b78b2a-6676-4572-9b89-7b8e60df80f6\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") "
Nov 02 14:06:50.231464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231400    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-certs\") pod \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") "
Nov 02 14:06:50.231464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231436    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58b78b2a-6676-4572-9b89-7b8e60df80f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b78b2a-6676-4572-9b89-7b8e60df80f6-istiod-ca-cert\") pod \"58b78b2a-6676-4572-9b89-7b8e60df80f6\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") "
Nov 02 14:06:50.231625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231484    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-podinfo\") pod \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") "
Nov 02 14:06:50.231625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231532    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-certs\") pod \"58b78b2a-6676-4572-9b89-7b8e60df80f6\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") "
Nov 02 14:06:50.231625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231585    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-token\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-token\") pod \"58b78b2a-6676-4572-9b89-7b8e60df80f6\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") "
Nov 02 14:06:50.231830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231645    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-envoy\") pod \"58b78b2a-6676-4572-9b89-7b8e60df80f6\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") "
Nov 02 14:06:50.231830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231675    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-socket\") pod \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") "
Nov 02 14:06:50.231830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231725    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-podinfo\") pod \"58b78b2a-6676-4572-9b89-7b8e60df80f6\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") "
Nov 02 14:06:50.231830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231768    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-kube-api-access-vsjg6\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-kube-api-access-vsjg6\") pod \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") "
Nov 02 14:06:50.232064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231863    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-credential-socket\") pod \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") "
Nov 02 14:06:50.232064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231933    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-kube-api-access-xnm8w\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-kube-api-access-xnm8w\") pod \"58b78b2a-6676-4572-9b89-7b8e60df80f6\" (UID: \"58b78b2a-6676-4572-9b89-7b8e60df80f6\") "
Nov 02 14:06:50.232064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.231973    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-data\") pod \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\" (UID: \"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4\") "
Nov 02 14:06:50.232228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.232110    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-certs" pod "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" (UID: "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.232476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.232450    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istiod-ca-cert" pod "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" (UID: "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:50.232711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.232689    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-data" pod "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" (UID: "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.233018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.232997    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.233018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.233006    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58b78b2a-6676-4572-9b89-7b8e60df80f6-istiod-ca-cert" pod "58b78b2a-6676-4572-9b89-7b8e60df80f6" (UID: "58b78b2a-6676-4572-9b89-7b8e60df80f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:06:50.233161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.233023    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.233161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.233040    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.234823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.234794    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-socket" pod "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" (UID: "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.235082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.235057    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-certs" pod "58b78b2a-6676-4572-9b89-7b8e60df80f6" (UID: "58b78b2a-6676-4572-9b89-7b8e60df80f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.236173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.235984    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-credential-socket" pod "58b78b2a-6676-4572-9b89-7b8e60df80f6" (UID: "58b78b2a-6676-4572-9b89-7b8e60df80f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.236173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.236002    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-credential-socket" pod "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" (UID: "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.236331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.236285    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-socket" pod "58b78b2a-6676-4572-9b89-7b8e60df80f6" (UID: "58b78b2a-6676-4572-9b89-7b8e60df80f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.236400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.236350    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-data" pod "58b78b2a-6676-4572-9b89-7b8e60df80f6" (UID: "58b78b2a-6676-4572-9b89-7b8e60df80f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.237769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.237741    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-envoy" pod "58b78b2a-6676-4572-9b89-7b8e60df80f6" (UID: "58b78b2a-6676-4572-9b89-7b8e60df80f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.239227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.239204    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-token" pod "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" (UID: "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:50.239463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.239327    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-kube-api-access-xnm8w" pod "58b78b2a-6676-4572-9b89-7b8e60df80f6" (UID: "58b78b2a-6676-4572-9b89-7b8e60df80f6"). InnerVolumeSpecName "kube-api-access-xnm8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:50.239463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.239430    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-kube-api-access-vsjg6" pod "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" (UID: "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4"). InnerVolumeSpecName "kube-api-access-vsjg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:50.239821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.239755    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-envoy" pod "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" (UID: "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:06:50.240048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.239997    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-podinfo" pod "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" (UID: "ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:06:50.240286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.240263    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-token" pod "58b78b2a-6676-4572-9b89-7b8e60df80f6" (UID: "58b78b2a-6676-4572-9b89-7b8e60df80f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:06:50.240828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.240803    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-podinfo" pod "58b78b2a-6676-4572-9b89-7b8e60df80f6" (UID: "58b78b2a-6676-4572-9b89-7b8e60df80f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:06:50.335377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335341    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335370    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335396    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335405    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335414    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335424    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58b78b2a-6676-4572-9b89-7b8e60df80f6-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335432    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335439    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335448    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335467    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335477    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335485    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58b78b2a-6676-4572-9b89-7b8e60df80f6-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335494    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsjg6\" (UniqueName: \"kubernetes.io/projected/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-kube-api-access-vsjg6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335502    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.335614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.335512    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnm8w\" (UniqueName: \"kubernetes.io/projected/58b78b2a-6676-4572-9b89-7b8e60df80f6-kube-api-access-xnm8w\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:06:50.708012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.707974    5034 generic.go:356] "Generic (PLEG): container finished" podID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerID="337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612" exitCode=0
Nov 02 14:06:50.708012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.708014    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"
Nov 02 14:06:50.708230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.708053    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" event={"ID":"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4","Type":"ContainerDied","Data":"337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612"}
Nov 02 14:06:50.708230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.708108    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp" event={"ID":"ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4","Type":"ContainerDied","Data":"37f9eb6d25e3ff257c8264e16c2492b16dd437d7e0e4fcd51877ae8275e9781d"}
Nov 02 14:06:50.708230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.708131    5034 scope.go:122] "RemoveContainer" containerID="337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612"
Nov 02 14:06:50.710033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.710007    5034 generic.go:356] "Generic (PLEG): container finished" podID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerID="db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df" exitCode=0
Nov 02 14:06:50.710159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.710065    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"
Nov 02 14:06:50.710159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.710097    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" event={"ID":"58b78b2a-6676-4572-9b89-7b8e60df80f6","Type":"ContainerDied","Data":"db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df"}
Nov 02 14:06:50.710287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.710163    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj" event={"ID":"58b78b2a-6676-4572-9b89-7b8e60df80f6","Type":"ContainerDied","Data":"0a34b0e1286ac4be3e29f39d4b4d565433ca52df8f7d7baf196cd95ac59e95ca"}
Nov 02 14:06:50.722109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.721856    5034 scope.go:122] "RemoveContainer" containerID="337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612"
Nov 02 14:06:50.722109 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:06:50.722094    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612\": container with ID starting with 337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612 not found: ID does not exist" containerID="337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612"
Nov 02 14:06:50.722242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.722125    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612"} err="failed to get container status \"337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612\": rpc error: code = NotFound desc = could not find container \"337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612\": container with ID starting with 337afcdda2863809ce8b314928c4324c8efada1d38aa82eab07dcd8c0e95f612 not found: ID does not exist"
Nov 02 14:06:50.722242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.722149    5034 scope.go:122] "RemoveContainer" containerID="db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df"
Nov 02 14:06:50.731845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.731827    5034 scope.go:122] "RemoveContainer" containerID="db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df"
Nov 02 14:06:50.732110 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:06:50.732093    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df\": container with ID starting with db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df not found: ID does not exist" containerID="db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df"
Nov 02 14:06:50.732186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.732117    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df"} err="failed to get container status \"db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df\": rpc error: code = NotFound desc = could not find container \"db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df\": container with ID starting with db3b8daf2f3f29eac80e8960564d4afd2367dee8be848694870647ee815c76df not found: ID does not exist"
Nov 02 14:06:50.766790 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.766761    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"]
Nov 02 14:06:50.796566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.796537    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-mpxkj"]
Nov 02 14:06:50.826182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.826159    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"]
Nov 02 14:06:50.830688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:50.830653    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-98zzp"]
Nov 02 14:06:51.402173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:51.402141    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" path="/var/lib/kubelet/pods/58b78b2a-6676-4572-9b89-7b8e60df80f6/volumes"
Nov 02 14:06:51.402684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:51.402670    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" path="/var/lib/kubelet/pods/ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4/volumes"
Nov 02 14:06:55.495972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:55.495931    5034 patch_prober.go:29] interesting pod/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:06:55.496419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:55.496009    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:06:57.631312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:57.631277    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:06:57.812885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:57.812852    5034 generic.go:356] "Generic (PLEG): container finished" podID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerID="eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc" exitCode=0
Nov 02 14:06:57.813092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:57.812927    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" event={"ID":"b22ed9db-6d6c-4575-bff2-881a07c7aa04","Type":"ContainerDied","Data":"eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc"}
Nov 02 14:06:57.813424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:57.813403    5034 scope.go:122] "RemoveContainer" containerID="eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc"
Nov 02 14:06:57.815137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:57.815105    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:57.815267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:57.815172    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:57.815267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:57.815205    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:06:58.825623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:58.825259    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" event={"ID":"b22ed9db-6d6c-4575-bff2-881a07c7aa04","Type":"ContainerStarted","Data":"50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae"}
Nov 02 14:06:58.988196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:58.988160    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:06:58.988723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:58.988698    5034 patch_prober.go:29] interesting pod/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 14:06:58.988932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:58.988902    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 14:06:59.038799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:59.038747    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"]
Nov 02 14:06:59.044556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:59.044523    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2cd0adc-nsftw"]
Nov 02 14:06:59.414668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:59.414633    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f1a70e-b375-49cc-b456-cf86a19733b2" path="/var/lib/kubelet/pods/f7f1a70e-b375-49cc-b456-cf86a19733b2/volumes"
Nov 02 14:07:00.000804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:59.995179    5034 patch_prober.go:29] interesting pod/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 14:07:00.000804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:06:59.995241    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 14:07:00.990801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:00.989872    5034 patch_prober.go:29] interesting pod/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 14:07:00.990801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:00.989938    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 14:07:01.981811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:01.980125    5034 patch_prober.go:29] interesting pod/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 14:07:01.981811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:01.980203    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 14:07:02.982803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:02.979489    5034 patch_prober.go:29] interesting pod/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 14:07:02.982803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:02.979552    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 14:07:03.981804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:03.980370    5034 patch_prober.go:29] interesting pod/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body=
Nov 02 14:07:03.981804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:03.980428    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused"
Nov 02 14:07:05.016023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:05.015140    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:07:05.038803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:05.019730    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:07:05.056831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:05.039289    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:07:12.694927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:12.694888    5034 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:07:12.696563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:12.694973    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:07:12.972294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:12.972196    5034 generic.go:356] "Generic (PLEG): container finished" podID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerID="2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4" exitCode=0
Nov 02 14:07:12.972294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:12.972271    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" event={"ID":"abf07384-0c83-4ffd-94d8-a448e809bf6c","Type":"ContainerDied","Data":"2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4"}
Nov 02 14:07:12.972732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:12.972709    5034 scope.go:122] "RemoveContainer" containerID="2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4"
Nov 02 14:07:12.974489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:12.974459    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:07:12.974598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:12.974543    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:07:12.974662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:12.974597    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:07:14.002803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:14.002305    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" event={"ID":"abf07384-0c83-4ffd-94d8-a448e809bf6c","Type":"ContainerStarted","Data":"21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5"}
Nov 02 14:07:14.203802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:14.203124    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:07:14.203802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:14.203443    5034 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 14:07:14.203802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:14.203500    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 14:07:15.202627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:15.202591    5034 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 14:07:15.203177 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:15.203148    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 14:07:16.205803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:16.205187    5034 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 14:07:16.205803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:16.205248    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 14:07:17.203435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:17.203393    5034 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 14:07:17.203654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:17.203470    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 14:07:18.202530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:18.202487    5034 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 14:07:18.203184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:18.203134    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 14:07:18.453312 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:07:18.453217    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:07:19.213803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:19.213257    5034 patch_prober.go:29] interesting pod/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Nov 02 14:07:19.213803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:19.213337    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Nov 02 14:07:20.132879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.132834    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"]
Nov 02 14:07:20.134444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.134417    5034 cpu_manager.go:414] "containerMap: removing container" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy"
Nov 02 14:07:20.134633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.134619    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy"
Nov 02 14:07:20.134767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.134756    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy"
Nov 02 14:07:20.134879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.134867    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy"
Nov 02 14:07:20.135257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.135243    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="58b78b2a-6676-4572-9b89-7b8e60df80f6" containerName="istio-proxy"
Nov 02 14:07:20.135380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.135370    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2ff075-0f93-4dc6-86f4-5b0e2a50b4e4" containerName="istio-proxy"
Nov 02 14:07:20.143091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.143070    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:20.198990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.198954    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-794667-dockercfg-qq5hc\""
Nov 02 14:07:20.216852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.199591    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-794667\""
Nov 02 14:07:20.249868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.227179    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e33df8-c051-420e-ba83-d86f6a253dc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8\" (UID: \"92e33df8-c051-420e-ba83-d86f6a253dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:20.250568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.250530    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8vh\" (UniqueName: \"kubernetes.io/projected/92e33df8-c051-420e-ba83-d86f6a253dc6-kube-api-access-zj8vh\") pod \"payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8\" (UID: \"92e33df8-c051-420e-ba83-d86f6a253dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:20.288821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.265207    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:07:20.293085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.291534    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:07:20.314942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.314890    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:07:20.339476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.339442    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"]
Nov 02 14:07:20.352180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.352139    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"]
Nov 02 14:07:20.359375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.359345    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:20.361933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.361903    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e33df8-c051-420e-ba83-d86f6a253dc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8\" (UID: \"92e33df8-c051-420e-ba83-d86f6a253dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:20.362479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.362451    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8vh\" (UniqueName: \"kubernetes.io/projected/92e33df8-c051-420e-ba83-d86f6a253dc6-kube-api-access-zj8vh\") pod \"payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8\" (UID: \"92e33df8-c051-420e-ba83-d86f6a253dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:20.363658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.363632    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e33df8-c051-420e-ba83-d86f6a253dc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8\" (UID: \"92e33df8-c051-420e-ba83-d86f6a253dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:20.364450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.364430    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"]
Nov 02 14:07:20.417348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.417237    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8vh\" (UniqueName: \"kubernetes.io/projected/92e33df8-c051-420e-ba83-d86f6a253dc6-kube-api-access-zj8vh\") pod \"payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8\" (UID: \"92e33df8-c051-420e-ba83-d86f6a253dc6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:20.463905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.463865    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc1483e-0366-4104-90ea-3c1cbaa0de70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt\" (UID: \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:20.465184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.465155    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzg8s\" (UniqueName: \"kubernetes.io/projected/3dc1483e-0366-4104-90ea-3c1cbaa0de70-kube-api-access-dzg8s\") pod \"payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt\" (UID: \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:20.567329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.567297    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzg8s\" (UniqueName: \"kubernetes.io/projected/3dc1483e-0366-4104-90ea-3c1cbaa0de70-kube-api-access-dzg8s\") pod \"payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt\" (UID: \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:20.567640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.567624    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc1483e-0366-4104-90ea-3c1cbaa0de70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt\" (UID: \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:20.568628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.568602    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc1483e-0366-4104-90ea-3c1cbaa0de70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt\" (UID: \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:20.606121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.606088    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzg8s\" (UniqueName: \"kubernetes.io/projected/3dc1483e-0366-4104-90ea-3c1cbaa0de70-kube-api-access-dzg8s\") pod \"payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt\" (UID: \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:20.636966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.636933    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:20.721025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:20.720944    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:21.029391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:21.027772    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"]
Nov 02 14:07:21.029576 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:07:21.029497    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92e33df8_c051_420e_ba83_d86f6a253dc6.slice/crio-c08c6c184bc3b248025708bfa9a8d271d3f715cae35a8bc91b2202d49dc96b9a WatchSource:0}: Error finding container c08c6c184bc3b248025708bfa9a8d271d3f715cae35a8bc91b2202d49dc96b9a: Status 404 returned error can't find the container with id c08c6c184bc3b248025708bfa9a8d271d3f715cae35a8bc91b2202d49dc96b9a
Nov 02 14:07:21.104956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:21.104929    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"]
Nov 02 14:07:21.119245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:21.119192    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8" event={"ID":"92e33df8-c051-420e-ba83-d86f6a253dc6","Type":"ContainerStarted","Data":"c08c6c184bc3b248025708bfa9a8d271d3f715cae35a8bc91b2202d49dc96b9a"}
Nov 02 14:07:22.116459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:22.116339    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"]
Nov 02 14:07:22.135185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:22.135138    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt" event={"ID":"3dc1483e-0366-4104-90ea-3c1cbaa0de70","Type":"ContainerStarted","Data":"6d334c5c56d862a88e6f4228f9bc90929de952edb8b8f74d0f1651b9302d3daf"}
Nov 02 14:07:22.135441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:22.135418    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt" event={"ID":"3dc1483e-0366-4104-90ea-3c1cbaa0de70","Type":"ContainerStarted","Data":"b31f4f820abaf03927cb7f111db50a72b7c599e273590463618c60d3fb0376fc"}
Nov 02 14:07:22.135928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:22.135811    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:22.138755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:22.138728    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8" event={"ID":"92e33df8-c051-420e-ba83-d86f6a253dc6","Type":"ContainerStarted","Data":"a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999"}
Nov 02 14:07:22.138880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:22.138790    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:22.253821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:22.250844    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt" podStartSLOduration=2.250823315 podStartE2EDuration="2.250823315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:22.249073452 +0000 UTC m=+1433.000377904" watchObservedRunningTime="2026-11-02 14:07:22.250823315 +0000 UTC m=+1433.002127754"
Nov 02 14:07:22.349199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:22.349136    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8" podStartSLOduration=2.349112208 podStartE2EDuration="2.349112208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:07:22.342239882 +0000 UTC m=+1433.093544321" watchObservedRunningTime="2026-11-02 14:07:22.349112208 +0000 UTC m=+1433.100416650"
Nov 02 14:07:24.268680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:24.268650    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"]
Nov 02 14:07:24.298273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:24.298245    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35ac7333-4gslh"]
Nov 02 14:07:25.397510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:25.397475    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eca4eb5-319a-4629-8586-fd69b86c64c7" path="/var/lib/kubelet/pods/4eca4eb5-319a-4629-8586-fd69b86c64c7/volumes"
Nov 02 14:07:27.117454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:27.117391    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc" podUID="0f450776-6c08-461b-866b-aa2ade15fabd" containerName="payload-processing" containerID="cri-o://4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f" gracePeriod=25
Nov 02 14:07:27.450283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:27.450255    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:07:27.480131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:27.480096    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f450776-6c08-461b-866b-aa2ade15fabd-kube-api-access-xgsgp\" (UniqueName: \"kubernetes.io/projected/0f450776-6c08-461b-866b-aa2ade15fabd-kube-api-access-xgsgp\") pod \"0f450776-6c08-461b-866b-aa2ade15fabd\" (UID: \"0f450776-6c08-461b-866b-aa2ade15fabd\") "
Nov 02 14:07:27.480319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:27.480219    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f450776-6c08-461b-866b-aa2ade15fabd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f450776-6c08-461b-866b-aa2ade15fabd-plugins-config-volume\") pod \"0f450776-6c08-461b-866b-aa2ade15fabd\" (UID: \"0f450776-6c08-461b-866b-aa2ade15fabd\") "
Nov 02 14:07:27.480772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:27.480727    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f450776-6c08-461b-866b-aa2ade15fabd-plugins-config-volume" pod "0f450776-6c08-461b-866b-aa2ade15fabd" (UID: "0f450776-6c08-461b-866b-aa2ade15fabd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:07:27.483004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:27.482979    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f450776-6c08-461b-866b-aa2ade15fabd-kube-api-access-xgsgp" pod "0f450776-6c08-461b-866b-aa2ade15fabd" (UID: "0f450776-6c08-461b-866b-aa2ade15fabd"). InnerVolumeSpecName "kube-api-access-xgsgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:07:27.582379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:27.582333    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f450776-6c08-461b-866b-aa2ade15fabd-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:07:27.582379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:27.582378    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgsgp\" (UniqueName: \"kubernetes.io/projected/0f450776-6c08-461b-866b-aa2ade15fabd-kube-api-access-xgsgp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:07:28.189209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.189173    5034 generic.go:356] "Generic (PLEG): container finished" podID="0f450776-6c08-461b-866b-aa2ade15fabd" containerID="4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f" exitCode=0
Nov 02 14:07:28.189650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.189242    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"
Nov 02 14:07:28.189650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.189265    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc" event={"ID":"0f450776-6c08-461b-866b-aa2ade15fabd","Type":"ContainerDied","Data":"4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f"}
Nov 02 14:07:28.189650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.189322    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc" event={"ID":"0f450776-6c08-461b-866b-aa2ade15fabd","Type":"ContainerDied","Data":"3bcb6f16928a170f23c6500f6dd54faa775ba0d0c2e9bf1147065be03716bdd2"}
Nov 02 14:07:28.189650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.189353    5034 scope.go:122] "RemoveContainer" containerID="4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f"
Nov 02 14:07:28.201773 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.201756    5034 scope.go:122] "RemoveContainer" containerID="4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f"
Nov 02 14:07:28.202090 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:07:28.202066    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f\": container with ID starting with 4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f not found: ID does not exist" containerID="4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f"
Nov 02 14:07:28.202200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.202093    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f"} err="failed to get container status \"4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f\": rpc error: code = NotFound desc = could not find container \"4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f\": container with ID starting with 4f3df8355c8555d806485779cdb4b728a4eaf7955f4ed704a6bf38976a49835f not found: ID does not exist"
Nov 02 14:07:28.251916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.251886    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"]
Nov 02 14:07:28.264648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.264624    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c7d2df-7d986d4b87-pw7mc"]
Nov 02 14:07:28.363900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.363868    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:07:28.364064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:28.363919    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:07:29.245277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:29.245237    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"]
Nov 02 14:07:29.398406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:29.398379    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f450776-6c08-461b-866b-aa2ade15fabd" path="/var/lib/kubelet/pods/0f450776-6c08-461b-866b-aa2ade15fabd/volumes"
Nov 02 14:07:30.209393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:30.209356    5034 generic.go:356] "Generic (PLEG): container finished" podID="15c57591-6c30-490a-b348-e97fbbd80319" containerID="dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc" exitCode=0
Nov 02 14:07:30.209583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:30.209426    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" event={"ID":"15c57591-6c30-490a-b348-e97fbbd80319","Type":"ContainerDied","Data":"dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc"}
Nov 02 14:07:30.209955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:30.209938    5034 scope.go:122] "RemoveContainer" containerID="dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc"
Nov 02 14:07:30.212006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:30.211973    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:07:30.212110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:30.212064    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:07:30.212150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:30.212129    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:07:31.221357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:31.221326    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" event={"ID":"15c57591-6c30-490a-b348-e97fbbd80319","Type":"ContainerStarted","Data":"07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413"}
Nov 02 14:07:31.523770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:31.523734    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:07:31.524015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:31.523918    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:07:31.524015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:31.523971    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 14:07:32.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:32.523715    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:07:32.524229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:32.524204    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 14:07:33.152805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:33.150961    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:07:33.559448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:33.559408    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:07:33.559859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:33.559485    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 14:07:34.156770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.156740    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:07:34.246503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.246464    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f" podUID="394d2a81-57f2-4c32-989f-492a4595968c" containerName="payload-pre-processing" containerID="cri-o://44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba" gracePeriod=25
Nov 02 14:07:34.524869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.524289    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:07:34.524869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.524354    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 14:07:34.652446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.652424    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:07:34.740543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.740512    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394d2a81-57f2-4c32-989f-492a4595968c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394d2a81-57f2-4c32-989f-492a4595968c-plugins-config-volume\") pod \"394d2a81-57f2-4c32-989f-492a4595968c\" (UID: \"394d2a81-57f2-4c32-989f-492a4595968c\") "
Nov 02 14:07:34.740724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.740590    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394d2a81-57f2-4c32-989f-492a4595968c-kube-api-access-xp29f\" (UniqueName: \"kubernetes.io/projected/394d2a81-57f2-4c32-989f-492a4595968c-kube-api-access-xp29f\") pod \"394d2a81-57f2-4c32-989f-492a4595968c\" (UID: \"394d2a81-57f2-4c32-989f-492a4595968c\") "
Nov 02 14:07:34.741024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.740992    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394d2a81-57f2-4c32-989f-492a4595968c-plugins-config-volume" pod "394d2a81-57f2-4c32-989f-492a4595968c" (UID: "394d2a81-57f2-4c32-989f-492a4595968c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:07:34.741329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.741294    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/394d2a81-57f2-4c32-989f-492a4595968c-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:07:34.742860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.742839    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394d2a81-57f2-4c32-989f-492a4595968c-kube-api-access-xp29f" pod "394d2a81-57f2-4c32-989f-492a4595968c" (UID: "394d2a81-57f2-4c32-989f-492a4595968c"). InnerVolumeSpecName "kube-api-access-xp29f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:07:34.842438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:34.842355    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp29f\" (UniqueName: \"kubernetes.io/projected/394d2a81-57f2-4c32-989f-492a4595968c-kube-api-access-xp29f\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:07:35.210850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.210740    5034 scope.go:122] "RemoveContainer" containerID="1f05aa7b629a4d2edc37bfd9b231a3eab36320eac12bc57e84c26d16a02dacee"
Nov 02 14:07:35.222275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.222255    5034 scope.go:122] "RemoveContainer" containerID="c97b8d5d80ee7da1495756d7832c85c526519d63c2ca9998f07d8b069bf7fea1"
Nov 02 14:07:35.231863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.231832    5034 scope.go:122] "RemoveContainer" containerID="ae48816bebe372492ce321a89b81b51ee6e22bbfc8544cafcbf22cb00a2aaa63"
Nov 02 14:07:35.241356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.241339    5034 scope.go:122] "RemoveContainer" containerID="d8536952f9c1d19f47bc9c16f7aa567f7d302d3d04088dd7507133dfa8c7795d"
Nov 02 14:07:35.250658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.250502    5034 scope.go:122] "RemoveContainer" containerID="df1a858da76e1b3475b1e7f8f5b290f856fd68de96ee92545fed872e9f9c17c9"
Nov 02 14:07:35.331029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.330996    5034 generic.go:356] "Generic (PLEG): container finished" podID="394d2a81-57f2-4c32-989f-492a4595968c" containerID="44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba" exitCode=0
Nov 02 14:07:35.331191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.331171    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f" event={"ID":"394d2a81-57f2-4c32-989f-492a4595968c","Type":"ContainerDied","Data":"44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba"}
Nov 02 14:07:35.331236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.331226    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f" event={"ID":"394d2a81-57f2-4c32-989f-492a4595968c","Type":"ContainerDied","Data":"f6e49d680b77ed2b2d7558514b94044191247063b0c1b224f3d07203df3492e1"}
Nov 02 14:07:35.331271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.331255    5034 scope.go:122] "RemoveContainer" containerID="44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba"
Nov 02 14:07:35.331407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.331394    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"
Nov 02 14:07:35.346750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.346603    5034 scope.go:122] "RemoveContainer" containerID="44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba"
Nov 02 14:07:35.347190 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:07:35.347108    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba\": container with ID starting with 44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba not found: ID does not exist" containerID="44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba"
Nov 02 14:07:35.347190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.347136    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba"} err="failed to get container status \"44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba\": rpc error: code = NotFound desc = could not find container \"44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba\": container with ID starting with 44ffca3009f30f672b99aa7133f70ea5fa76536303b4f48345ba254bb99b13ba not found: ID does not exist"
Nov 02 14:07:35.416563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.416537    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"]
Nov 02 14:07:35.420442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.420419    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b7489f-67b9746999-lmw8f"]
Nov 02 14:07:35.524801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.524454    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:07:35.524801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:35.524500    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 14:07:36.524640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:36.524605    5034 patch_prober.go:29] interesting pod/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Nov 02 14:07:36.525152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:36.524670    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Nov 02 14:07:37.395604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:37.395560    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394d2a81-57f2-4c32-989f-492a4595968c" path="/var/lib/kubelet/pods/394d2a81-57f2-4c32-989f-492a4595968c/volumes"
Nov 02 14:07:37.529369 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:37.529338    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:07:37.529918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:37.529879    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:07:37.530692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:37.530669    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:07:40.692202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:40.692168    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"]
Nov 02 14:07:40.726231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:40.726197    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"]
Nov 02 14:07:45.693956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:45.693916    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5" podUID="ffb5e6e8-b98a-437f-8799-e47bf23e98aa" containerName="payload-processing" containerID="cri-o://cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196" gracePeriod=25
Nov 02 14:07:45.726667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:45.726600    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858" podUID="b9c24e50-85aa-4321-b00b-fa9de3b3aa55" containerName="payload-pre-processing" containerID="cri-o://74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584" gracePeriod=25
Nov 02 14:07:46.204993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.204606    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:07:46.322226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.322195    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:07:46.341365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.341339    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-kube-api-access-nxhgq\" (UniqueName: \"kubernetes.io/projected/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-kube-api-access-nxhgq\") pod \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\" (UID: \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\") "
Nov 02 14:07:46.341568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.341549    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-plugins-config-volume\") pod \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\" (UID: \"ffb5e6e8-b98a-437f-8799-e47bf23e98aa\") "
Nov 02 14:07:46.342016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.341989    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-plugins-config-volume" pod "ffb5e6e8-b98a-437f-8799-e47bf23e98aa" (UID: "ffb5e6e8-b98a-437f-8799-e47bf23e98aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:07:46.342270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.342251    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:07:46.343561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.343541    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-kube-api-access-nxhgq" pod "ffb5e6e8-b98a-437f-8799-e47bf23e98aa" (UID: "ffb5e6e8-b98a-437f-8799-e47bf23e98aa"). InnerVolumeSpecName "kube-api-access-nxhgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:07:46.442705 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.442672    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-plugins-config-volume\") pod \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\" (UID: \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\") "
Nov 02 14:07:46.442923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.442882    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-kube-api-access-8jh6r\" (UniqueName: \"kubernetes.io/projected/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-kube-api-access-8jh6r\") pod \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\" (UID: \"b9c24e50-85aa-4321-b00b-fa9de3b3aa55\") "
Nov 02 14:07:46.443146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.443121    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-plugins-config-volume" pod "b9c24e50-85aa-4321-b00b-fa9de3b3aa55" (UID: "b9c24e50-85aa-4321-b00b-fa9de3b3aa55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:07:46.443503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.443481    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:07:46.443589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.443509    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxhgq\" (UniqueName: \"kubernetes.io/projected/ffb5e6e8-b98a-437f-8799-e47bf23e98aa-kube-api-access-nxhgq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:07:46.445033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.445012    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-kube-api-access-8jh6r" pod "b9c24e50-85aa-4321-b00b-fa9de3b3aa55" (UID: "b9c24e50-85aa-4321-b00b-fa9de3b3aa55"). InnerVolumeSpecName "kube-api-access-8jh6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:07:46.449955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.449924    5034 generic.go:356] "Generic (PLEG): container finished" podID="b9c24e50-85aa-4321-b00b-fa9de3b3aa55" containerID="74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584" exitCode=0
Nov 02 14:07:46.450040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.449982    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"
Nov 02 14:07:46.450040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.450003    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858" event={"ID":"b9c24e50-85aa-4321-b00b-fa9de3b3aa55","Type":"ContainerDied","Data":"74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584"}
Nov 02 14:07:46.450118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.450058    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858" event={"ID":"b9c24e50-85aa-4321-b00b-fa9de3b3aa55","Type":"ContainerDied","Data":"98e349dc361f6ac877d778692609055a7e6f3f6e3cceeabb73ee903d15e372df"}
Nov 02 14:07:46.450118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.450083    5034 scope.go:122] "RemoveContainer" containerID="74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584"
Nov 02 14:07:46.451413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.451394    5034 generic.go:356] "Generic (PLEG): container finished" podID="ffb5e6e8-b98a-437f-8799-e47bf23e98aa" containerID="cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196" exitCode=0
Nov 02 14:07:46.451498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.451433    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"
Nov 02 14:07:46.451498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.451477    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5" event={"ID":"ffb5e6e8-b98a-437f-8799-e47bf23e98aa","Type":"ContainerDied","Data":"cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196"}
Nov 02 14:07:46.451631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.451522    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5" event={"ID":"ffb5e6e8-b98a-437f-8799-e47bf23e98aa","Type":"ContainerDied","Data":"12e8833efc88d43b14d22058676343ab4282a092f1899b79b4949efb68b1975f"}
Nov 02 14:07:46.460276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.460256    5034 scope.go:122] "RemoveContainer" containerID="74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584"
Nov 02 14:07:46.460531 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:07:46.460511    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584\": container with ID starting with 74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584 not found: ID does not exist" containerID="74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584"
Nov 02 14:07:46.460616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.460543    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584"} err="failed to get container status \"74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584\": rpc error: code = NotFound desc = could not find container \"74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584\": container with ID starting with 74db3ae4bcf566d56a7fb835023bc2ad9de8572d450c56738b6d97a9d3204584 not found: ID does not exist"
Nov 02 14:07:46.460616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.460566    5034 scope.go:122] "RemoveContainer" containerID="cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196"
Nov 02 14:07:46.471492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.471474    5034 scope.go:122] "RemoveContainer" containerID="cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196"
Nov 02 14:07:46.471752 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:07:46.471729    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196\": container with ID starting with cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196 not found: ID does not exist" containerID="cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196"
Nov 02 14:07:46.471966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.471760    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196"} err="failed to get container status \"cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196\": rpc error: code = NotFound desc = could not find container \"cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196\": container with ID starting with cdb1ced721232be5f75049fb60f5a5153b7ef788c4f54742fd034d4e3ce7c196 not found: ID does not exist"
Nov 02 14:07:46.512182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.512157    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"]
Nov 02 14:07:46.534320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.534295    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f66ca3-5d95bc475f-l9858"]
Nov 02 14:07:46.545080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.545059    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jh6r\" (UniqueName: \"kubernetes.io/projected/b9c24e50-85aa-4321-b00b-fa9de3b3aa55-kube-api-access-8jh6r\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:07:46.575884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.575825    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"]
Nov 02 14:07:46.583369 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:46.583346    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f66ca3-6879bb6695-65fd5"]
Nov 02 14:07:47.395352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:47.395319    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c24e50-85aa-4321-b00b-fa9de3b3aa55" path="/var/lib/kubelet/pods/b9c24e50-85aa-4321-b00b-fa9de3b3aa55/volumes"
Nov 02 14:07:47.395844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:47.395829    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb5e6e8-b98a-437f-8799-e47bf23e98aa" path="/var/lib/kubelet/pods/ffb5e6e8-b98a-437f-8799-e47bf23e98aa/volumes"
Nov 02 14:07:48.518949 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:07:48.518910    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:07:50.043330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:50.043296    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"]
Nov 02 14:07:50.059103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:50.059063    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c9af55-pnb9x"]
Nov 02 14:07:51.395702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:07:51.395665    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06467082-aef9-4454-9f96-c8f21c174b9b" path="/var/lib/kubelet/pods/06467082-aef9-4454-9f96-c8f21c174b9b/volumes"
Nov 02 14:08:07.917584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:07.917540    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"]
Nov 02 14:08:07.918274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:07.917822    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr" podUID="074235ec-9cde-4e11-b5bf-0160a6c5729a" containerName="payload-processing" containerID="cri-o://258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3" gracePeriod=30
Nov 02 14:08:08.261978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.261952    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:08:08.294058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.293894    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074235ec-9cde-4e11-b5bf-0160a6c5729a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074235ec-9cde-4e11-b5bf-0160a6c5729a-plugins-config-volume\") pod \"074235ec-9cde-4e11-b5bf-0160a6c5729a\" (UID: \"074235ec-9cde-4e11-b5bf-0160a6c5729a\") "
Nov 02 14:08:08.294058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.293942    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074235ec-9cde-4e11-b5bf-0160a6c5729a-kube-api-access-lpctn\" (UniqueName: \"kubernetes.io/projected/074235ec-9cde-4e11-b5bf-0160a6c5729a-kube-api-access-lpctn\") pod \"074235ec-9cde-4e11-b5bf-0160a6c5729a\" (UID: \"074235ec-9cde-4e11-b5bf-0160a6c5729a\") "
Nov 02 14:08:08.296300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.296257    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074235ec-9cde-4e11-b5bf-0160a6c5729a-plugins-config-volume" pod "074235ec-9cde-4e11-b5bf-0160a6c5729a" (UID: "074235ec-9cde-4e11-b5bf-0160a6c5729a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:08.402807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.401154    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/074235ec-9cde-4e11-b5bf-0160a6c5729a-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:08.402807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.402731    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074235ec-9cde-4e11-b5bf-0160a6c5729a-kube-api-access-lpctn" pod "074235ec-9cde-4e11-b5bf-0160a6c5729a" (UID: "074235ec-9cde-4e11-b5bf-0160a6c5729a"). InnerVolumeSpecName "kube-api-access-lpctn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:08.502336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.502307    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpctn\" (UniqueName: \"kubernetes.io/projected/074235ec-9cde-4e11-b5bf-0160a6c5729a-kube-api-access-lpctn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:08.655845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.654370    5034 generic.go:356] "Generic (PLEG): container finished" podID="074235ec-9cde-4e11-b5bf-0160a6c5729a" containerID="258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3" exitCode=0
Nov 02 14:08:08.655845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.654592    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr" event={"ID":"074235ec-9cde-4e11-b5bf-0160a6c5729a","Type":"ContainerDied","Data":"258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3"}
Nov 02 14:08:08.655845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.654644    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr" event={"ID":"074235ec-9cde-4e11-b5bf-0160a6c5729a","Type":"ContainerDied","Data":"321123769b4b70fcff44cb5140f51073ab43c6090e08ba754185f3ed5ef3fe9e"}
Nov 02 14:08:08.655845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.654672    5034 scope.go:122] "RemoveContainer" containerID="258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3"
Nov 02 14:08:08.655845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.654903    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"
Nov 02 14:08:08.680771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.680747    5034 scope.go:122] "RemoveContainer" containerID="258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3"
Nov 02 14:08:08.681123 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:08.681092    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3\": container with ID starting with 258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3 not found: ID does not exist" containerID="258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3"
Nov 02 14:08:08.681205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.681134    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3"} err="failed to get container status \"258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3\": rpc error: code = NotFound desc = could not find container \"258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3\": container with ID starting with 258216e5d7567db0c43b0195949e3d6c5bde5d51c8a58750c08b114f9d3c44b3 not found: ID does not exist"
Nov 02 14:08:08.722126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.722097    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"]
Nov 02 14:08:08.728329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:08.728305    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fb206f-7c976b8f5b-84swr"]
Nov 02 14:08:09.396470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:09.396432    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074235ec-9cde-4e11-b5bf-0160a6c5729a" path="/var/lib/kubelet/pods/074235ec-9cde-4e11-b5bf-0160a6c5729a/volumes"
Nov 02 14:08:09.673596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:09.673242    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"]
Nov 02 14:08:09.673596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:09.673491    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy" containerID="cri-o://21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5" gracePeriod=30
Nov 02 14:08:14.299235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:14.299146    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"]
Nov 02 14:08:14.299713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:14.299597    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy" containerID="cri-o://144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f" gracePeriod=30
Nov 02 14:08:15.017122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.017098    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:08:15.159107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159077    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-kube-api-access-dljnj\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-kube-api-access-dljnj\") pod \"abf07384-0c83-4ffd-94d8-a448e809bf6c\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") "
Nov 02 14:08:15.159262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159129    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-socket\") pod \"abf07384-0c83-4ffd-94d8-a448e809bf6c\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") "
Nov 02 14:08:15.159262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159194    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-token\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-token\") pod \"abf07384-0c83-4ffd-94d8-a448e809bf6c\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") "
Nov 02 14:08:15.159262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159220    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-data\") pod \"abf07384-0c83-4ffd-94d8-a448e809bf6c\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") "
Nov 02 14:08:15.159262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159247    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-envoy\") pod \"abf07384-0c83-4ffd-94d8-a448e809bf6c\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") "
Nov 02 14:08:15.159492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159277    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-certs\") pod \"abf07384-0c83-4ffd-94d8-a448e809bf6c\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") "
Nov 02 14:08:15.159492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159396    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf07384-0c83-4ffd-94d8-a448e809bf6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf07384-0c83-4ffd-94d8-a448e809bf6c-istiod-ca-cert\") pod \"abf07384-0c83-4ffd-94d8-a448e809bf6c\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") "
Nov 02 14:08:15.159492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159429    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-podinfo\") pod \"abf07384-0c83-4ffd-94d8-a448e809bf6c\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") "
Nov 02 14:08:15.159492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159442    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-socket" pod "abf07384-0c83-4ffd-94d8-a448e809bf6c" (UID: "abf07384-0c83-4ffd-94d8-a448e809bf6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.159492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159467    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-credential-socket\") pod \"abf07384-0c83-4ffd-94d8-a448e809bf6c\" (UID: \"abf07384-0c83-4ffd-94d8-a448e809bf6c\") "
Nov 02 14:08:15.159743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159623    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-certs" pod "abf07384-0c83-4ffd-94d8-a448e809bf6c" (UID: "abf07384-0c83-4ffd-94d8-a448e809bf6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.159743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159637    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-data" pod "abf07384-0c83-4ffd-94d8-a448e809bf6c" (UID: "abf07384-0c83-4ffd-94d8-a448e809bf6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.159864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159827    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf07384-0c83-4ffd-94d8-a448e809bf6c-istiod-ca-cert" pod "abf07384-0c83-4ffd-94d8-a448e809bf6c" (UID: "abf07384-0c83-4ffd-94d8-a448e809bf6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:15.159932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.159907    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-credential-socket" pod "abf07384-0c83-4ffd-94d8-a448e809bf6c" (UID: "abf07384-0c83-4ffd-94d8-a448e809bf6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.160415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.160390    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.160415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.160415    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.160586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.160426    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.160586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.160435    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf07384-0c83-4ffd-94d8-a448e809bf6c-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.160586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.160442    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.161734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.161708    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-podinfo" pod "abf07384-0c83-4ffd-94d8-a448e809bf6c" (UID: "abf07384-0c83-4ffd-94d8-a448e809bf6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:08:15.162072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.162052    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-envoy" pod "abf07384-0c83-4ffd-94d8-a448e809bf6c" (UID: "abf07384-0c83-4ffd-94d8-a448e809bf6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:15.162215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.162195    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-kube-api-access-dljnj" pod "abf07384-0c83-4ffd-94d8-a448e809bf6c" (UID: "abf07384-0c83-4ffd-94d8-a448e809bf6c"). InnerVolumeSpecName "kube-api-access-dljnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:15.162281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.162223    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-token" pod "abf07384-0c83-4ffd-94d8-a448e809bf6c" (UID: "abf07384-0c83-4ffd-94d8-a448e809bf6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:15.261432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.261393    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.261432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.261422    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dljnj\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-kube-api-access-dljnj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.261582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.261437    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.261582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.261460    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf07384-0c83-4ffd-94d8-a448e809bf6c-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:15.729059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.728976    5034 generic.go:356] "Generic (PLEG): container finished" podID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerID="21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5" exitCode=0
Nov 02 14:08:15.729516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.729193    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"
Nov 02 14:08:15.729516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.729257    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" event={"ID":"abf07384-0c83-4ffd-94d8-a448e809bf6c","Type":"ContainerDied","Data":"21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5"}
Nov 02 14:08:15.729516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.729340    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v" event={"ID":"abf07384-0c83-4ffd-94d8-a448e809bf6c","Type":"ContainerDied","Data":"9c144e4a34d1894ebfb7e66121e1793aee65418c4338fcc1a1538133bee82a00"}
Nov 02 14:08:15.729516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.729375    5034 scope.go:122] "RemoveContainer" containerID="21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5"
Nov 02 14:08:15.740629 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.740609    5034 scope.go:122] "RemoveContainer" containerID="2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4"
Nov 02 14:08:15.751066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.751050    5034 scope.go:122] "RemoveContainer" containerID="21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5"
Nov 02 14:08:15.751414 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:15.751383    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5\": container with ID starting with 21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5 not found: ID does not exist" containerID="21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5"
Nov 02 14:08:15.751556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.751421    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5"} err="failed to get container status \"21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5\": rpc error: code = NotFound desc = could not find container \"21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5\": container with ID starting with 21a457dffed2c9f57724691e42bf23dca4f1305df0c2ea9a5aad3b632b537bf5 not found: ID does not exist"
Nov 02 14:08:15.751556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.751446    5034 scope.go:122] "RemoveContainer" containerID="2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4"
Nov 02 14:08:15.751717 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:15.751689    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4\": container with ID starting with 2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4 not found: ID does not exist" containerID="2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4"
Nov 02 14:08:15.751860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.751722    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4"} err="failed to get container status \"2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4\": rpc error: code = NotFound desc = could not find container \"2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4\": container with ID starting with 2b5140d2555d4e5728b637a58b6fd36a95660bd6bdc7d5496b60e8375384f0b4 not found: ID does not exist"
Nov 02 14:08:15.810223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.810197    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"]
Nov 02 14:08:15.823067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:15.823044    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f66ca3-openshift-default-6dd664dc97-6kg4v"]
Nov 02 14:08:16.368649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.368612    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"]
Nov 02 14:08:16.369605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369583    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ffb5e6e8-b98a-437f-8799-e47bf23e98aa" containerName="payload-processing"
Nov 02 14:08:16.369707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369608    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb5e6e8-b98a-437f-8799-e47bf23e98aa" containerName="payload-processing"
Nov 02 14:08:16.369707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369664    5034 cpu_manager.go:414] "containerMap: removing container" podUID="074235ec-9cde-4e11-b5bf-0160a6c5729a" containerName="payload-processing"
Nov 02 14:08:16.369707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369672    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074235ec-9cde-4e11-b5bf-0160a6c5729a" containerName="payload-processing"
Nov 02 14:08:16.369707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369701    5034 cpu_manager.go:414] "containerMap: removing container" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy"
Nov 02 14:08:16.369912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369709    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy"
Nov 02 14:08:16.369912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369723    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b9c24e50-85aa-4321-b00b-fa9de3b3aa55" containerName="payload-pre-processing"
Nov 02 14:08:16.369912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369731    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c24e50-85aa-4321-b00b-fa9de3b3aa55" containerName="payload-pre-processing"
Nov 02 14:08:16.369912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369748    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0f450776-6c08-461b-866b-aa2ade15fabd" containerName="payload-processing"
Nov 02 14:08:16.369912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369755    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f450776-6c08-461b-866b-aa2ade15fabd" containerName="payload-processing"
Nov 02 14:08:16.369912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369766    5034 cpu_manager.go:414] "containerMap: removing container" podUID="394d2a81-57f2-4c32-989f-492a4595968c" containerName="payload-pre-processing"
Nov 02 14:08:16.369912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369795    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394d2a81-57f2-4c32-989f-492a4595968c" containerName="payload-pre-processing"
Nov 02 14:08:16.403272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.369984    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb5e6e8-b98a-437f-8799-e47bf23e98aa" containerName="payload-processing"
Nov 02 14:08:16.403272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.370003    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="074235ec-9cde-4e11-b5bf-0160a6c5729a" containerName="payload-processing"
Nov 02 14:08:16.403272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.370016    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy"
Nov 02 14:08:16.403272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.370026    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c24e50-85aa-4321-b00b-fa9de3b3aa55" containerName="payload-pre-processing"
Nov 02 14:08:16.403272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.370039    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f450776-6c08-461b-866b-aa2ade15fabd" containerName="payload-processing"
Nov 02 14:08:16.403272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.370047    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="394d2a81-57f2-4c32-989f-492a4595968c" containerName="payload-pre-processing"
Nov 02 14:08:16.403272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.370055    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy"
Nov 02 14:08:16.403272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.379658    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.403272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.396894    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b7d400-openshift-default-dockercfg-xwzht\""
Nov 02 14:08:16.417638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.417608    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"]
Nov 02 14:08:16.483920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.483884    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-istio-token\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.484079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.483944    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5048905b-cee0-4c96-ab97-a91a67d749db-istiod-ca-cert\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.484079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.484002    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-certs\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.484079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.484047    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-socket\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.484079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.484073    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-data\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.484224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.484120    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqszt\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-kube-api-access-cqszt\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.484309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.484281    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-credential-socket\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.484440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.484375    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5048905b-cee0-4c96-ab97-a91a67d749db-istio-podinfo\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.484555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.484537    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-envoy\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.592801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.591764    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-istio-token\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.592801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.591856    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5048905b-cee0-4c96-ab97-a91a67d749db-istiod-ca-cert\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.592801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.591919    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-certs\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.592801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.591975    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-socket\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.592801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.592001    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-data\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.592801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.592045    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqszt\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-kube-api-access-cqszt\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.592801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.592174    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-credential-socket\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.592801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.592232    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5048905b-cee0-4c96-ab97-a91a67d749db-istio-podinfo\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.592801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.592273    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-envoy\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.593434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.593143    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-socket\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.608810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.594040    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5048905b-cee0-4c96-ab97-a91a67d749db-istiod-ca-cert\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.608810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.594261    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-certs\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.608810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.594695    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-data\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.608810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.595110    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-credential-socket\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.608810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.606314    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-envoy\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.622796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.619397    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-istio-token\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.647179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.624693    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqszt\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-kube-api-access-cqszt\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.647179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.636358    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5048905b-cee0-4c96-ab97-a91a67d749db-istio-podinfo\") pod \"e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") " pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:16.720800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:16.715458    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:17.088940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.088913    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"]
Nov 02 14:08:17.092090 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:08:17.092063    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5048905b_cee0_4c96_ab97_a91a67d749db.slice/crio-398428c4626c4d688f0709ee3f453a916753e1710d4a75459e5a1c62a05e6232 WatchSource:0}: Error finding container 398428c4626c4d688f0709ee3f453a916753e1710d4a75459e5a1c62a05e6232: Status 404 returned error can't find the container with id 398428c4626c4d688f0709ee3f453a916753e1710d4a75459e5a1c62a05e6232
Nov 02 14:08:17.094786 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.094741    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:08:17.094877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.094847    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:08:17.094920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.094881    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:08:17.402203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.402120    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" path="/var/lib/kubelet/pods/abf07384-0c83-4ffd-94d8-a448e809bf6c/volumes"
Nov 02 14:08:17.869496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.856907    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"]
Nov 02 14:08:17.869496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.857892    5034 cpu_manager.go:414] "containerMap: removing container" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy"
Nov 02 14:08:17.869496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.857913    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf07384-0c83-4ffd-94d8-a448e809bf6c" containerName="istio-proxy"
Nov 02 14:08:17.869496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.868640    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:17.891794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.888311    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-13367a-dockercfg-f57jx\""
Nov 02 14:08:17.891794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.889438    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-13367a\""
Nov 02 14:08:17.905140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.904820    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"]
Nov 02 14:08:17.909405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.909366    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" event={"ID":"5048905b-cee0-4c96-ab97-a91a67d749db","Type":"ContainerStarted","Data":"990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45"}
Nov 02 14:08:17.909612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.909593    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" event={"ID":"5048905b-cee0-4c96-ab97-a91a67d749db","Type":"ContainerStarted","Data":"398428c4626c4d688f0709ee3f453a916753e1710d4a75459e5a1c62a05e6232"}
Nov 02 14:08:17.966083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:17.966023    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" podStartSLOduration=1.966006123 podStartE2EDuration="1.966006123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:17.961412074 +0000 UTC m=+1488.712716513" watchObservedRunningTime="2026-11-02 14:08:17.966006123 +0000 UTC m=+1488.717310561"
Nov 02 14:08:18.068145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.068105    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d68081-4a0b-4e95-9e37-c5c8718cc589-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6\" (UID: \"72d68081-4a0b-4e95-9e37-c5c8718cc589\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:18.068527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.068487    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd97n\" (UniqueName: \"kubernetes.io/projected/72d68081-4a0b-4e95-9e37-c5c8718cc589-kube-api-access-xd97n\") pod \"payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6\" (UID: \"72d68081-4a0b-4e95-9e37-c5c8718cc589\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:18.169856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.169642    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd97n\" (UniqueName: \"kubernetes.io/projected/72d68081-4a0b-4e95-9e37-c5c8718cc589-kube-api-access-xd97n\") pod \"payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6\" (UID: \"72d68081-4a0b-4e95-9e37-c5c8718cc589\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:18.169856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.169742    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d68081-4a0b-4e95-9e37-c5c8718cc589-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6\" (UID: \"72d68081-4a0b-4e95-9e37-c5c8718cc589\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:18.186797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.171383    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d68081-4a0b-4e95-9e37-c5c8718cc589-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6\" (UID: \"72d68081-4a0b-4e95-9e37-c5c8718cc589\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:18.229799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.229630    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd97n\" (UniqueName: \"kubernetes.io/projected/72d68081-4a0b-4e95-9e37-c5c8718cc589-kube-api-access-xd97n\") pod \"payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6\" (UID: \"72d68081-4a0b-4e95-9e37-c5c8718cc589\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:18.414212 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:18.414176    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:08:18.514581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.514550    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:18.717462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.717430    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:18.723413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.723388    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:18.827161 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:08:18.827129    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d68081_4a0b_4e95_9e37_c5c8718cc589.slice/crio-2947607379dfb0b55e8edfce852fb6c2b86f51dc709d0222a5213dad002c3e60 WatchSource:0}: Error finding container 2947607379dfb0b55e8edfce852fb6c2b86f51dc709d0222a5213dad002c3e60: Status 404 returned error can't find the container with id 2947607379dfb0b55e8edfce852fb6c2b86f51dc709d0222a5213dad002c3e60
Nov 02 14:08:18.830032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.830009    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"]
Nov 02 14:08:18.919387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.919359    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6" event={"ID":"72d68081-4a0b-4e95-9e37-c5c8718cc589","Type":"ContainerStarted","Data":"2947607379dfb0b55e8edfce852fb6c2b86f51dc709d0222a5213dad002c3e60"}
Nov 02 14:08:18.919679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.919661    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:18.920625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:18.920605    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:08:19.608895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.608869    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:08:19.704130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704091    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-podinfo\") pod \"9109731a-74f9-4294-92f3-dc6da573a3c8\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") "
Nov 02 14:08:19.704309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704201    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-certs\") pod \"9109731a-74f9-4294-92f3-dc6da573a3c8\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") "
Nov 02 14:08:19.704309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704260    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-token\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-token\") pod \"9109731a-74f9-4294-92f3-dc6da573a3c8\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") "
Nov 02 14:08:19.704436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704325    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-envoy\") pod \"9109731a-74f9-4294-92f3-dc6da573a3c8\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") "
Nov 02 14:08:19.704491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704455    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-credential-socket\") pod \"9109731a-74f9-4294-92f3-dc6da573a3c8\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") "
Nov 02 14:08:19.704564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704537    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-certs" pod "9109731a-74f9-4294-92f3-dc6da573a3c8" (UID: "9109731a-74f9-4294-92f3-dc6da573a3c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:19.704626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704581    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-kube-api-access-hrvrx\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-kube-api-access-hrvrx\") pod \"9109731a-74f9-4294-92f3-dc6da573a3c8\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") "
Nov 02 14:08:19.704688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704674    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9109731a-74f9-4294-92f3-dc6da573a3c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9109731a-74f9-4294-92f3-dc6da573a3c8-istiod-ca-cert\") pod \"9109731a-74f9-4294-92f3-dc6da573a3c8\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") "
Nov 02 14:08:19.704821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704749    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-credential-socket" pod "9109731a-74f9-4294-92f3-dc6da573a3c8" (UID: "9109731a-74f9-4294-92f3-dc6da573a3c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:19.704887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704837    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-socket\") pod \"9109731a-74f9-4294-92f3-dc6da573a3c8\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") "
Nov 02 14:08:19.704991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.704961    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-data\") pod \"9109731a-74f9-4294-92f3-dc6da573a3c8\" (UID: \"9109731a-74f9-4294-92f3-dc6da573a3c8\") "
Nov 02 14:08:19.705195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.705151    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9109731a-74f9-4294-92f3-dc6da573a3c8-istiod-ca-cert" pod "9109731a-74f9-4294-92f3-dc6da573a3c8" (UID: "9109731a-74f9-4294-92f3-dc6da573a3c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:19.705394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.705375    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-socket" pod "9109731a-74f9-4294-92f3-dc6da573a3c8" (UID: "9109731a-74f9-4294-92f3-dc6da573a3c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:19.705568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.705548    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-data" pod "9109731a-74f9-4294-92f3-dc6da573a3c8" (UID: "9109731a-74f9-4294-92f3-dc6da573a3c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:19.705758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.705714    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9109731a-74f9-4294-92f3-dc6da573a3c8-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:19.705895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.705758    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:19.705895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.705773    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:19.705895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.705807    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:19.705895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.705819    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:19.706949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.706919    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-envoy" pod "9109731a-74f9-4294-92f3-dc6da573a3c8" (UID: "9109731a-74f9-4294-92f3-dc6da573a3c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:19.707189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.707161    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-kube-api-access-hrvrx" pod "9109731a-74f9-4294-92f3-dc6da573a3c8" (UID: "9109731a-74f9-4294-92f3-dc6da573a3c8"). InnerVolumeSpecName "kube-api-access-hrvrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:19.708045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.708019    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-token" pod "9109731a-74f9-4294-92f3-dc6da573a3c8" (UID: "9109731a-74f9-4294-92f3-dc6da573a3c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:19.708137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.708104    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-podinfo" pod "9109731a-74f9-4294-92f3-dc6da573a3c8" (UID: "9109731a-74f9-4294-92f3-dc6da573a3c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:08:19.807904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.807872    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:19.807904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.807905    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:19.808132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.807919    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9109731a-74f9-4294-92f3-dc6da573a3c8-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:19.808132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.807932    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrvrx\" (UniqueName: \"kubernetes.io/projected/9109731a-74f9-4294-92f3-dc6da573a3c8-kube-api-access-hrvrx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:19.929748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.929712    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6" event={"ID":"72d68081-4a0b-4e95-9e37-c5c8718cc589","Type":"ContainerStarted","Data":"652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739"}
Nov 02 14:08:19.929969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.929822    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:19.931552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.931521    5034 generic.go:356] "Generic (PLEG): container finished" podID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerID="144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f" exitCode=0
Nov 02 14:08:19.931666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.931553    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" event={"ID":"9109731a-74f9-4294-92f3-dc6da573a3c8","Type":"ContainerDied","Data":"144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f"}
Nov 02 14:08:19.931666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.931568    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"
Nov 02 14:08:19.931666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.931594    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k" event={"ID":"9109731a-74f9-4294-92f3-dc6da573a3c8","Type":"ContainerDied","Data":"ff8127e4d50f2c6f1065941b8732ca84983b6407a3e2c5786360957beecd7188"}
Nov 02 14:08:19.931666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.931615    5034 scope.go:122] "RemoveContainer" containerID="144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f"
Nov 02 14:08:19.942243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.942226    5034 scope.go:122] "RemoveContainer" containerID="a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de"
Nov 02 14:08:19.952642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.952623    5034 scope.go:122] "RemoveContainer" containerID="144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f"
Nov 02 14:08:19.952914 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:19.952890    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f\": container with ID starting with 144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f not found: ID does not exist" containerID="144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f"
Nov 02 14:08:19.953010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.952923    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f"} err="failed to get container status \"144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f\": rpc error: code = NotFound desc = could not find container \"144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f\": container with ID starting with 144cb7ae52fd3fff3ebd7c150a194d3ce97e50f6751a3d5e907f1d01a6c6a90f not found: ID does not exist"
Nov 02 14:08:19.953010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.952944    5034 scope.go:122] "RemoveContainer" containerID="a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de"
Nov 02 14:08:19.953195 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:19.953177    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de\": container with ID starting with a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de not found: ID does not exist" containerID="a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de"
Nov 02 14:08:19.953233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.953200    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de"} err="failed to get container status \"a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de\": rpc error: code = NotFound desc = could not find container \"a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de\": container with ID starting with a16999ad8df30463a4e3d795d1e413faf882049fadd731eb7b83f061707f00de not found: ID does not exist"
Nov 02 14:08:19.975134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:19.975088    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6" podStartSLOduration=2.9750769630000002 podStartE2EDuration="2.975076963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:19.97225117 +0000 UTC m=+1490.723555622" watchObservedRunningTime="2026-11-02 14:08:19.975076963 +0000 UTC m=+1490.726381422"
Nov 02 14:08:20.014737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:20.014712    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"]
Nov 02 14:08:20.020001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:20.019982    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fb206f-openshift-default-7fb49b77cc-f289k"]
Nov 02 14:08:21.395549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:21.395514    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" path="/var/lib/kubelet/pods/9109731a-74f9-4294-92f3-dc6da573a3c8/volumes"
Nov 02 14:08:22.378013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.377980    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"]
Nov 02 14:08:22.378639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.378623    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy"
Nov 02 14:08:22.378639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.378638    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy"
Nov 02 14:08:22.378832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.378675    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy"
Nov 02 14:08:22.378832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.378680    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy"
Nov 02 14:08:22.378832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.378815    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy"
Nov 02 14:08:22.382272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.382251    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:22.400717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.400685    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b7d400-dockercfg-z2qlz\""
Nov 02 14:08:22.404793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.404751    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"]
Nov 02 14:08:22.411793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.411760    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b7d400\""
Nov 02 14:08:22.443763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.443734    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"]
Nov 02 14:08:22.444627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.444594    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7pz\" (UniqueName: \"kubernetes.io/projected/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-kube-api-access-rq7pz\") pod \"payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2\" (UID: \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:22.444764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.444667    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2\" (UID: \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:22.445557 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.445527    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9109731a-74f9-4294-92f3-dc6da573a3c8" containerName="istio-proxy"
Nov 02 14:08:22.449848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.448733    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:22.498026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.491748    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"]
Nov 02 14:08:22.572864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.560391    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk2qp\" (UniqueName: \"kubernetes.io/projected/d13007cc-d6c8-447a-9653-d40f74514064-kube-api-access-wk2qp\") pod \"payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv\" (UID: \"d13007cc-d6c8-447a-9653-d40f74514064\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:22.572864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.560554    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13007cc-d6c8-447a-9653-d40f74514064-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv\" (UID: \"d13007cc-d6c8-447a-9653-d40f74514064\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:22.572864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.560665    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7pz\" (UniqueName: \"kubernetes.io/projected/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-kube-api-access-rq7pz\") pod \"payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2\" (UID: \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:22.572864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.560725    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2\" (UID: \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:22.572864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.561719    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2\" (UID: \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:22.628692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.628602    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7pz\" (UniqueName: \"kubernetes.io/projected/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-kube-api-access-rq7pz\") pod \"payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2\" (UID: \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:22.672806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.671079    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk2qp\" (UniqueName: \"kubernetes.io/projected/d13007cc-d6c8-447a-9653-d40f74514064-kube-api-access-wk2qp\") pod \"payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv\" (UID: \"d13007cc-d6c8-447a-9653-d40f74514064\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:22.672806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.671239    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13007cc-d6c8-447a-9653-d40f74514064-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv\" (UID: \"d13007cc-d6c8-447a-9653-d40f74514064\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:22.672806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.672344    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13007cc-d6c8-447a-9653-d40f74514064-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv\" (UID: \"d13007cc-d6c8-447a-9653-d40f74514064\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:22.705798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.702064    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:22.717811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.713392    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk2qp\" (UniqueName: \"kubernetes.io/projected/d13007cc-d6c8-447a-9653-d40f74514064-kube-api-access-wk2qp\") pod \"payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv\" (UID: \"d13007cc-d6c8-447a-9653-d40f74514064\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:22.796501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:22.795045    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:23.075407 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:08:23.075373    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4633fd_49f3_4cf4_8457_c89cc30aa4dd.slice/crio-c71d440bd3620f89a228b936658de2b902c0787e04907dad60eeafb8587effd1 WatchSource:0}: Error finding container c71d440bd3620f89a228b936658de2b902c0787e04907dad60eeafb8587effd1: Status 404 returned error can't find the container with id c71d440bd3620f89a228b936658de2b902c0787e04907dad60eeafb8587effd1
Nov 02 14:08:23.076713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:23.076681    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"]
Nov 02 14:08:23.149520 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:08:23.149495    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13007cc_d6c8_447a_9653_d40f74514064.slice/crio-46962d4dc9055c2cef9309b8a2315d01a70373b0c81d4196710d83ba4df15d6c WatchSource:0}: Error finding container 46962d4dc9055c2cef9309b8a2315d01a70373b0c81d4196710d83ba4df15d6c: Status 404 returned error can't find the container with id 46962d4dc9055c2cef9309b8a2315d01a70373b0c81d4196710d83ba4df15d6c
Nov 02 14:08:23.151186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:23.151164    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"]
Nov 02 14:08:23.978911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:23.967536    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2" event={"ID":"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd","Type":"ContainerStarted","Data":"e8eb0a3c04fb4c7ec6d33926d1dea3a1c971427d4e89ad95842b231a1c5f6b79"}
Nov 02 14:08:23.978911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:23.967618    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2" event={"ID":"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd","Type":"ContainerStarted","Data":"c71d440bd3620f89a228b936658de2b902c0787e04907dad60eeafb8587effd1"}
Nov 02 14:08:23.978911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:23.968365    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:23.978911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:23.969925    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv" event={"ID":"d13007cc-d6c8-447a-9653-d40f74514064","Type":"ContainerStarted","Data":"629db5881abf8cebd6c393fd349e55218bfd80a3810f530110e95834bbfe4dfb"}
Nov 02 14:08:23.978911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:23.969970    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:23.978911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:23.969997    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv" event={"ID":"d13007cc-d6c8-447a-9653-d40f74514064","Type":"ContainerStarted","Data":"46962d4dc9055c2cef9309b8a2315d01a70373b0c81d4196710d83ba4df15d6c"}
Nov 02 14:08:24.038061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:24.037906    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2" podStartSLOduration=2.037888859 podStartE2EDuration="2.037888859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:24.037654522 +0000 UTC m=+1494.788958961" watchObservedRunningTime="2026-11-02 14:08:24.037888859 +0000 UTC m=+1494.789193298"
Nov 02 14:08:24.104381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:24.104326    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv" podStartSLOduration=2.104308614 podStartE2EDuration="2.104308614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:08:24.101338154 +0000 UTC m=+1494.852642593" watchObservedRunningTime="2026-11-02 14:08:24.104308614 +0000 UTC m=+1494.855613053"
Nov 02 14:08:29.005320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:29.005281    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"]
Nov 02 14:08:29.006096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:29.006052    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy" containerID="cri-o://368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7" gracePeriod=30
Nov 02 14:08:29.782430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:29.782385    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:08:29.790080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:29.790044    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:08:30.945626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:30.945594    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:08:34.319649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.319624    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:08:34.420143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420112    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-certs\") pod \"fae86fff-9730-4845-b60e-a0eac4259ef8\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") "
Nov 02 14:08:34.420306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420202    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-envoy\") pod \"fae86fff-9730-4845-b60e-a0eac4259ef8\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") "
Nov 02 14:08:34.420306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420262    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-podinfo\") pod \"fae86fff-9730-4845-b60e-a0eac4259ef8\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") "
Nov 02 14:08:34.420306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420297    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-socket\") pod \"fae86fff-9730-4845-b60e-a0eac4259ef8\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") "
Nov 02 14:08:34.420501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420338    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-data\") pod \"fae86fff-9730-4845-b60e-a0eac4259ef8\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") "
Nov 02 14:08:34.420501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420381    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-certs" pod "fae86fff-9730-4845-b60e-a0eac4259ef8" (UID: "fae86fff-9730-4845-b60e-a0eac4259ef8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:34.420501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420428    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-token\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-token\") pod \"fae86fff-9730-4845-b60e-a0eac4259ef8\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") "
Nov 02 14:08:34.420501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420476    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae86fff-9730-4845-b60e-a0eac4259ef8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae86fff-9730-4845-b60e-a0eac4259ef8-istiod-ca-cert\") pod \"fae86fff-9730-4845-b60e-a0eac4259ef8\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") "
Nov 02 14:08:34.420700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420564    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-credential-socket\") pod \"fae86fff-9730-4845-b60e-a0eac4259ef8\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") "
Nov 02 14:08:34.420700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420656    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-socket" pod "fae86fff-9730-4845-b60e-a0eac4259ef8" (UID: "fae86fff-9730-4845-b60e-a0eac4259ef8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:34.420700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420688    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-kube-api-access-gnhzx\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-kube-api-access-gnhzx\") pod \"fae86fff-9730-4845-b60e-a0eac4259ef8\" (UID: \"fae86fff-9730-4845-b60e-a0eac4259ef8\") "
Nov 02 14:08:34.420998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.420969    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-credential-socket" pod "fae86fff-9730-4845-b60e-a0eac4259ef8" (UID: "fae86fff-9730-4845-b60e-a0eac4259ef8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:34.421085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.421063    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-data" pod "fae86fff-9730-4845-b60e-a0eac4259ef8" (UID: "fae86fff-9730-4845-b60e-a0eac4259ef8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:34.421463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.421437    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae86fff-9730-4845-b60e-a0eac4259ef8-istiod-ca-cert" pod "fae86fff-9730-4845-b60e-a0eac4259ef8" (UID: "fae86fff-9730-4845-b60e-a0eac4259ef8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:34.421585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.421501    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:34.421585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.421524    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:34.421585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.421538    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:34.421585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.421552    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:34.422823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.422798    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-token" pod "fae86fff-9730-4845-b60e-a0eac4259ef8" (UID: "fae86fff-9730-4845-b60e-a0eac4259ef8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:34.422961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.422945    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-envoy" pod "fae86fff-9730-4845-b60e-a0eac4259ef8" (UID: "fae86fff-9730-4845-b60e-a0eac4259ef8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:34.423025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.423002    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-podinfo" pod "fae86fff-9730-4845-b60e-a0eac4259ef8" (UID: "fae86fff-9730-4845-b60e-a0eac4259ef8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:08:34.423141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.423124    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-kube-api-access-gnhzx" pod "fae86fff-9730-4845-b60e-a0eac4259ef8" (UID: "fae86fff-9730-4845-b60e-a0eac4259ef8"). InnerVolumeSpecName "kube-api-access-gnhzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:34.523092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.523062    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnhzx\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-kube-api-access-gnhzx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:34.523092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.523086    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:34.523092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.523095    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:34.523302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.523104    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fae86fff-9730-4845-b60e-a0eac4259ef8-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:34.523302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.523112    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fae86fff-9730-4845-b60e-a0eac4259ef8-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:34.988221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.988196    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:08:34.988415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:34.988308    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:08:35.091371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.091147    5034 generic.go:356] "Generic (PLEG): container finished" podID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerID="368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7" exitCode=0
Nov 02 14:08:35.091371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.091225    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" event={"ID":"fae86fff-9730-4845-b60e-a0eac4259ef8","Type":"ContainerDied","Data":"368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7"}
Nov 02 14:08:35.091371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.091290    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj" event={"ID":"fae86fff-9730-4845-b60e-a0eac4259ef8","Type":"ContainerDied","Data":"9aba97f817f1d3ffa1a2729b38f8acd7087af4e62e91ace7182afa0e1c7bd106"}
Nov 02 14:08:35.091371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.091321    5034 scope.go:122] "RemoveContainer" containerID="368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7"
Nov 02 14:08:35.091609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.091458    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"
Nov 02 14:08:35.134182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.134154    5034 scope.go:122] "RemoveContainer" containerID="ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0"
Nov 02 14:08:35.144995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.144969    5034 scope.go:122] "RemoveContainer" containerID="368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7"
Nov 02 14:08:35.145310 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:35.145287    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7\": container with ID starting with 368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7 not found: ID does not exist" containerID="368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7"
Nov 02 14:08:35.145381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.145317    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7"} err="failed to get container status \"368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7\": rpc error: code = NotFound desc = could not find container \"368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7\": container with ID starting with 368128258e7a4dd4df5c583b4034bd5bf7ca849c0b1f89e04854e2c009d682d7 not found: ID does not exist"
Nov 02 14:08:35.145381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.145335    5034 scope.go:122] "RemoveContainer" containerID="ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0"
Nov 02 14:08:35.145625 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:35.145601    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0\": container with ID starting with ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0 not found: ID does not exist" containerID="ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0"
Nov 02 14:08:35.145738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.145627    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0"} err="failed to get container status \"ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0\": rpc error: code = NotFound desc = could not find container \"ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0\": container with ID starting with ae680b34a38a1e8300aa3342891cbb60150c22c9a505f78bae669a652cfcb8c0 not found: ID does not exist"
Nov 02 14:08:35.177575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.177545    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"]
Nov 02 14:08:35.183135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.183108    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-60ad62-openshift-default-677ffd6455-65njj"]
Nov 02 14:08:35.399109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.399026    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" path="/var/lib/kubelet/pods/fae86fff-9730-4845-b60e-a0eac4259ef8/volumes"
Nov 02 14:08:35.422168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.422142    5034 scope.go:122] "RemoveContainer" containerID="578c7eef180d9f7f8258fdbdf85980ab333af0c5d66bba1555f9760958bd23e2"
Nov 02 14:08:35.432307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.432287    5034 scope.go:122] "RemoveContainer" containerID="769e230ed6bc58cc1df6f80c60339d44645fc48c1d86b17f64d4499e950f6bd5"
Nov 02 14:08:35.444562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:35.444542    5034 scope.go:122] "RemoveContainer" containerID="26c3a6f41de3b4300ad8a70d6a32b2dc5bbd3117840ca734ee2562bd2cd7c0c9"
Nov 02 14:08:48.292029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:48.291987    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"]
Nov 02 14:08:48.292670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:48.292186    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy" containerID="cri-o://2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2" gracePeriod=30
Nov 02 14:08:48.305962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:48.305934    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"]
Nov 02 14:08:48.306168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:48.306145    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy" containerID="cri-o://a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1" gracePeriod=30
Nov 02 14:08:48.426824 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:48.426764    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:08:53.668804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.668460    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 14:08:53.761473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.761439    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-certs\") pod \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") "
Nov 02 14:08:53.761639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.761530    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-envoy\") pod \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") "
Nov 02 14:08:53.761639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.761613    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-data\") pod \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") "
Nov 02 14:08:53.761733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.761638    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-credential-socket\") pod \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") "
Nov 02 14:08:53.761733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.761683    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-token\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-token\") pod \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") "
Nov 02 14:08:53.761733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.761718    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-socket\") pod \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") "
Nov 02 14:08:53.761887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.761743    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istiod-ca-cert\") pod \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") "
Nov 02 14:08:53.761887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.761794    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-podinfo\") pod \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") "
Nov 02 14:08:53.761887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.761805    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-certs" pod "ae84b012-0de1-42cf-a1d3-da54f50e70a9" (UID: "ae84b012-0de1-42cf-a1d3-da54f50e70a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.762122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.762065    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-socket" pod "ae84b012-0de1-42cf-a1d3-da54f50e70a9" (UID: "ae84b012-0de1-42cf-a1d3-da54f50e70a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.762188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.762121    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-credential-socket" pod "ae84b012-0de1-42cf-a1d3-da54f50e70a9" (UID: "ae84b012-0de1-42cf-a1d3-da54f50e70a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.762502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.762474    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-data" pod "ae84b012-0de1-42cf-a1d3-da54f50e70a9" (UID: "ae84b012-0de1-42cf-a1d3-da54f50e70a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.763007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.762978    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istiod-ca-cert" pod "ae84b012-0de1-42cf-a1d3-da54f50e70a9" (UID: "ae84b012-0de1-42cf-a1d3-da54f50e70a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:53.764494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.764465    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-kube-api-access-j4x2s\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-kube-api-access-j4x2s\") pod \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\" (UID: \"ae84b012-0de1-42cf-a1d3-da54f50e70a9\") "
Nov 02 14:08:53.765421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.765395    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.765509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.765423    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.765509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.765437    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.765509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.765449    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.765509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.765461    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.766353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.766329    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-envoy" pod "ae84b012-0de1-42cf-a1d3-da54f50e70a9" (UID: "ae84b012-0de1-42cf-a1d3-da54f50e70a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.766427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.766345    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-podinfo" pod "ae84b012-0de1-42cf-a1d3-da54f50e70a9" (UID: "ae84b012-0de1-42cf-a1d3-da54f50e70a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:08:53.766427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.766368    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-token" pod "ae84b012-0de1-42cf-a1d3-da54f50e70a9" (UID: "ae84b012-0de1-42cf-a1d3-da54f50e70a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:53.766427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.766399    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-kube-api-access-j4x2s" pod "ae84b012-0de1-42cf-a1d3-da54f50e70a9" (UID: "ae84b012-0de1-42cf-a1d3-da54f50e70a9"). InnerVolumeSpecName "kube-api-access-j4x2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:53.791812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.787954    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 14:08:53.865910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.865880    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-credential-socket\") pod \"7cbb41e2-890d-4add-86b5-39d8581198d2\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") "
Nov 02 14:08:53.866112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.865965    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-socket\") pod \"7cbb41e2-890d-4add-86b5-39d8581198d2\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") "
Nov 02 14:08:53.866112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.865995    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-kube-api-access-z762t\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-kube-api-access-z762t\") pod \"7cbb41e2-890d-4add-86b5-39d8581198d2\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") "
Nov 02 14:08:53.866112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.866070    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-envoy\") pod \"7cbb41e2-890d-4add-86b5-39d8581198d2\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") "
Nov 02 14:08:53.866283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.866111    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-certs\") pod \"7cbb41e2-890d-4add-86b5-39d8581198d2\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") "
Nov 02 14:08:53.866283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.866143    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-data\") pod \"7cbb41e2-890d-4add-86b5-39d8581198d2\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") "
Nov 02 14:08:53.866283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.866204    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-podinfo\") pod \"7cbb41e2-890d-4add-86b5-39d8581198d2\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") "
Nov 02 14:08:53.866283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.866197    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-credential-socket" pod "7cbb41e2-890d-4add-86b5-39d8581198d2" (UID: "7cbb41e2-890d-4add-86b5-39d8581198d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.866283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.866249    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-socket" pod "7cbb41e2-890d-4add-86b5-39d8581198d2" (UID: "7cbb41e2-890d-4add-86b5-39d8581198d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.866537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.866354    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-token\") pod \"7cbb41e2-890d-4add-86b5-39d8581198d2\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") "
Nov 02 14:08:53.866537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.866458    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cbb41e2-890d-4add-86b5-39d8581198d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb41e2-890d-4add-86b5-39d8581198d2-istiod-ca-cert\") pod \"7cbb41e2-890d-4add-86b5-39d8581198d2\" (UID: \"7cbb41e2-890d-4add-86b5-39d8581198d2\") "
Nov 02 14:08:53.866770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.866744    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-data" pod "7cbb41e2-890d-4add-86b5-39d8581198d2" (UID: "7cbb41e2-890d-4add-86b5-39d8581198d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.867305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.867281    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.867417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.867322    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.867417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.867337    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.867417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.867351    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4x2s\" (UniqueName: \"kubernetes.io/projected/ae84b012-0de1-42cf-a1d3-da54f50e70a9-kube-api-access-j4x2s\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.867417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.867354    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-certs" pod "7cbb41e2-890d-4add-86b5-39d8581198d2" (UID: "7cbb41e2-890d-4add-86b5-39d8581198d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.867417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.867366    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae84b012-0de1-42cf-a1d3-da54f50e70a9-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.867417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.867381    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.867417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.867407    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.867693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.867502    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cbb41e2-890d-4add-86b5-39d8581198d2-istiod-ca-cert" pod "7cbb41e2-890d-4add-86b5-39d8581198d2" (UID: "7cbb41e2-890d-4add-86b5-39d8581198d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:08:53.922081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.921998    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-podinfo" pod "7cbb41e2-890d-4add-86b5-39d8581198d2" (UID: "7cbb41e2-890d-4add-86b5-39d8581198d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:08:53.922081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.922008    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-token" pod "7cbb41e2-890d-4add-86b5-39d8581198d2" (UID: "7cbb41e2-890d-4add-86b5-39d8581198d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:53.922081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.922059    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-envoy" pod "7cbb41e2-890d-4add-86b5-39d8581198d2" (UID: "7cbb41e2-890d-4add-86b5-39d8581198d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:08:53.922586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.922559    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-kube-api-access-z762t" pod "7cbb41e2-890d-4add-86b5-39d8581198d2" (UID: "7cbb41e2-890d-4add-86b5-39d8581198d2"). InnerVolumeSpecName "kube-api-access-z762t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:08:53.968575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.968547    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z762t\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-kube-api-access-z762t\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.968575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.968572    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.968575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.968581    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cbb41e2-890d-4add-86b5-39d8581198d2-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.968830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.968589    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.968830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.968597    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cbb41e2-890d-4add-86b5-39d8581198d2-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:53.968830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:53.968605    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cbb41e2-890d-4add-86b5-39d8581198d2-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:08:54.399801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.399750    5034 generic.go:356] "Generic (PLEG): container finished" podID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerID="2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2" exitCode=0
Nov 02 14:08:54.400147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.400125    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" event={"ID":"ae84b012-0de1-42cf-a1d3-da54f50e70a9","Type":"ContainerDied","Data":"2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2"}
Nov 02 14:08:54.400298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.400283    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq" event={"ID":"ae84b012-0de1-42cf-a1d3-da54f50e70a9","Type":"ContainerDied","Data":"0f217e206e33c6c0d3b73fffe352473e689b58ab03285f77938446e6e0203f0a"}
Nov 02 14:08:54.400403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.400391    5034 scope.go:122] "RemoveContainer" containerID="2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2"
Nov 02 14:08:54.400653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.400636    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"
Nov 02 14:08:54.415234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.409303    5034 generic.go:356] "Generic (PLEG): container finished" podID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerID="a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1" exitCode=0
Nov 02 14:08:54.415234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.409373    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" event={"ID":"7cbb41e2-890d-4add-86b5-39d8581198d2","Type":"ContainerDied","Data":"a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1"}
Nov 02 14:08:54.415234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.409412    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l" event={"ID":"7cbb41e2-890d-4add-86b5-39d8581198d2","Type":"ContainerDied","Data":"02ff1f81f036d53c07e20f8ad59a4530e5106010257ba1ecd792a5574ca83945"}
Nov 02 14:08:54.415234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.409538    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"
Nov 02 14:08:54.421807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.421489    5034 scope.go:122] "RemoveContainer" containerID="2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2"
Nov 02 14:08:54.422192 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:54.421838    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2\": container with ID starting with 2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2 not found: ID does not exist" containerID="2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2"
Nov 02 14:08:54.422192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.421872    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2"} err="failed to get container status \"2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2\": rpc error: code = NotFound desc = could not find container \"2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2\": container with ID starting with 2db0dd477e46dadbfe2f711cc902ce55a0bab1b65e354ec1d21d3454250941d2 not found: ID does not exist"
Nov 02 14:08:54.422192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.421893    5034 scope.go:122] "RemoveContainer" containerID="a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1"
Nov 02 14:08:54.431910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.431889    5034 scope.go:122] "RemoveContainer" containerID="a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1"
Nov 02 14:08:54.432162 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:08:54.432143    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1\": container with ID starting with a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1 not found: ID does not exist" containerID="a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1"
Nov 02 14:08:54.432215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.432168    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1"} err="failed to get container status \"a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1\": rpc error: code = NotFound desc = could not find container \"a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1\": container with ID starting with a8912571dc9022705d0c8971ee749013902c0067925e0b4d3e207575e1473cf1 not found: ID does not exist"
Nov 02 14:08:54.492080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.492046    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"]
Nov 02 14:08:54.505274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.505246    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-hdb7l"]
Nov 02 14:08:54.534867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.534826    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"]
Nov 02 14:08:54.538531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:54.538506    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-2rmgq"]
Nov 02 14:08:55.408668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:55.408631    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" path="/var/lib/kubelet/pods/7cbb41e2-890d-4add-86b5-39d8581198d2/volumes"
Nov 02 14:08:55.409346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:55.409324    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" path="/var/lib/kubelet/pods/ae84b012-0de1-42cf-a1d3-da54f50e70a9/volumes"
Nov 02 14:08:55.783860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:55.783823    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"]
Nov 02 14:08:55.784471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:55.784438    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy" containerID="cri-o://07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413" gracePeriod=30
Nov 02 14:08:59.062170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.062135    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"]
Nov 02 14:08:59.062905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.062878    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy"
Nov 02 14:08:59.062905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.062899    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy"
Nov 02 14:08:59.063111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.062925    5034 cpu_manager.go:414] "containerMap: removing container" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy"
Nov 02 14:08:59.063111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.062933    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy"
Nov 02 14:08:59.063111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.062956    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy"
Nov 02 14:08:59.063111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.062964    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy"
Nov 02 14:08:59.063321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.063171    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae84b012-0de1-42cf-a1d3-da54f50e70a9" containerName="istio-proxy"
Nov 02 14:08:59.063321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.063188    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy"
Nov 02 14:08:59.063321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.063201    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cbb41e2-890d-4add-86b5-39d8581198d2" containerName="istio-proxy"
Nov 02 14:08:59.063321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.063215    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy"
Nov 02 14:08:59.069045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.069022    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.092347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.092310    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-8acd34-openshift-default-dockercfg-6x2gq\""
Nov 02 14:08:59.118233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.118192    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"]
Nov 02 14:08:59.184180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.184131    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-certs\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.184376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.184243    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-envoy\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.184376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.184271    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-data\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.184376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.184311    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-podinfo\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.184376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.184350    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-credential-socket\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.184615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.184452    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99whh\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-kube-api-access-99whh\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.184615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.184521    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-token\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.184615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.184594    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60cfd239-3bd1-468f-81e1-bb85fee11656-istiod-ca-cert\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.184803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.184653    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-socket\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.285844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.285761    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-certs\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286129    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-envoy\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286170    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-data\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286208    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-podinfo\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286245    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-credential-socket\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286352    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99whh\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-kube-api-access-99whh\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286421    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-token\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286486    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-data\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286497    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60cfd239-3bd1-468f-81e1-bb85fee11656-istiod-ca-cert\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286581    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-socket\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.286769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286241    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-certs\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.288592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.286990    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-socket\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.288592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.287551    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-credential-socket\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.288825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.287842    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60cfd239-3bd1-468f-81e1-bb85fee11656-istiod-ca-cert\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.301605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.299061    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-envoy\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.302500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.302472    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-podinfo\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.313246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.313183    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-token\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.315987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.315960    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99whh\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-kube-api-access-99whh\") pod \"e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") " pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.434487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.434448    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:08:59.781558 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:08:59.781509    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60cfd239_3bd1_468f_81e1_bb85fee11656.slice/crio-61b30fffcf56e089a51352aabe5c71bd4322bd800b4f7273ebcdea7e02f42bae WatchSource:0}: Error finding container 61b30fffcf56e089a51352aabe5c71bd4322bd800b4f7273ebcdea7e02f42bae: Status 404 returned error can't find the container with id 61b30fffcf56e089a51352aabe5c71bd4322bd800b4f7273ebcdea7e02f42bae
Nov 02 14:08:59.782145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.782057    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"]
Nov 02 14:08:59.784734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.784692    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:08:59.784861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.784800    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:08:59.784861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:08:59.784844    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:00.491940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:00.491905    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" event={"ID":"60cfd239-3bd1-468f-81e1-bb85fee11656","Type":"ContainerStarted","Data":"9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308"}
Nov 02 14:09:00.492424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:00.491969    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" event={"ID":"60cfd239-3bd1-468f-81e1-bb85fee11656","Type":"ContainerStarted","Data":"61b30fffcf56e089a51352aabe5c71bd4322bd800b4f7273ebcdea7e02f42bae"}
Nov 02 14:09:00.542136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:00.542083    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podStartSLOduration=1.5420698590000002 podStartE2EDuration="1.542069859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:00.538347258 +0000 UTC m=+1531.289651897" watchObservedRunningTime="2026-11-02 14:09:00.542069859 +0000 UTC m=+1531.293374293"
Nov 02 14:09:01.117343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.117317    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:09:01.227317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.227281    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-istio-token\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-istio-token\") pod \"15c57591-6c30-490a-b348-e97fbbd80319\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") "
Nov 02 14:09:01.227522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.227351    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-certs\") pod \"15c57591-6c30-490a-b348-e97fbbd80319\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") "
Nov 02 14:09:01.227606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.227590    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c57591-6c30-490a-b348-e97fbbd80319-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c57591-6c30-490a-b348-e97fbbd80319-istiod-ca-cert\") pod \"15c57591-6c30-490a-b348-e97fbbd80319\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") "
Nov 02 14:09:01.227675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.227633    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-data\") pod \"15c57591-6c30-490a-b348-e97fbbd80319\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") "
Nov 02 14:09:01.227825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.227756    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-credential-socket\") pod \"15c57591-6c30-490a-b348-e97fbbd80319\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") "
Nov 02 14:09:01.228362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.227991    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-kube-api-access-s4bj2\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-kube-api-access-s4bj2\") pod \"15c57591-6c30-490a-b348-e97fbbd80319\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") "
Nov 02 14:09:01.228362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.228064    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-certs" pod "15c57591-6c30-490a-b348-e97fbbd80319" (UID: "15c57591-6c30-490a-b348-e97fbbd80319"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:09:01.228362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.228305    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-credential-socket" pod "15c57591-6c30-490a-b348-e97fbbd80319" (UID: "15c57591-6c30-490a-b348-e97fbbd80319"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:09:01.228362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.228322    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-data" pod "15c57591-6c30-490a-b348-e97fbbd80319" (UID: "15c57591-6c30-490a-b348-e97fbbd80319"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:09:01.228664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.228485    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c57591-6c30-490a-b348-e97fbbd80319-istiod-ca-cert" pod "15c57591-6c30-490a-b348-e97fbbd80319" (UID: "15c57591-6c30-490a-b348-e97fbbd80319"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:09:01.228764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.228743    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-socket\") pod \"15c57591-6c30-490a-b348-e97fbbd80319\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") "
Nov 02 14:09:01.228891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.228846    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-envoy\") pod \"15c57591-6c30-490a-b348-e97fbbd80319\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") "
Nov 02 14:09:01.229197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.229176    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-socket" pod "15c57591-6c30-490a-b348-e97fbbd80319" (UID: "15c57591-6c30-490a-b348-e97fbbd80319"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:09:01.230230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.229475    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15c57591-6c30-490a-b348-e97fbbd80319-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c57591-6c30-490a-b348-e97fbbd80319-istio-podinfo\") pod \"15c57591-6c30-490a-b348-e97fbbd80319\" (UID: \"15c57591-6c30-490a-b348-e97fbbd80319\") "
Nov 02 14:09:01.230230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.230127    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:01.230230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.230150    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:01.230230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.230168    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:01.230230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.230184    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c57591-6c30-490a-b348-e97fbbd80319-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:01.230230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.230197    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:01.231631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.231568    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-kube-api-access-s4bj2" pod "15c57591-6c30-490a-b348-e97fbbd80319" (UID: "15c57591-6c30-490a-b348-e97fbbd80319"). InnerVolumeSpecName "kube-api-access-s4bj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:09:01.231838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.231806    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-envoy" pod "15c57591-6c30-490a-b348-e97fbbd80319" (UID: "15c57591-6c30-490a-b348-e97fbbd80319"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:09:01.237827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.237247    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15c57591-6c30-490a-b348-e97fbbd80319-istio-podinfo" pod "15c57591-6c30-490a-b348-e97fbbd80319" (UID: "15c57591-6c30-490a-b348-e97fbbd80319"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:09:01.237827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.237366    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-istio-token" pod "15c57591-6c30-490a-b348-e97fbbd80319" (UID: "15c57591-6c30-490a-b348-e97fbbd80319"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:09:01.331983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.331949    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4bj2\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-kube-api-access-s4bj2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:01.331983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.331986    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c57591-6c30-490a-b348-e97fbbd80319-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:01.332150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.332000    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c57591-6c30-490a-b348-e97fbbd80319-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:01.332150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.332012    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c57591-6c30-490a-b348-e97fbbd80319-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:01.399247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.399172    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"]
Nov 02 14:09:01.435628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.435597    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:09:01.442289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.442265    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:09:01.544901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.544865    5034 generic.go:356] "Generic (PLEG): container finished" podID="15c57591-6c30-490a-b348-e97fbbd80319" containerID="07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413" exitCode=0
Nov 02 14:09:01.545406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.544937    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"
Nov 02 14:09:01.545406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.544952    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" event={"ID":"15c57591-6c30-490a-b348-e97fbbd80319","Type":"ContainerDied","Data":"07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413"}
Nov 02 14:09:01.545406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.545010    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd" event={"ID":"15c57591-6c30-490a-b348-e97fbbd80319","Type":"ContainerDied","Data":"1ce16cabe02289568b1e627e9cdd027df19188b1ec389a49452d1f2e2394204b"}
Nov 02 14:09:01.545406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.545043    5034 scope.go:122] "RemoveContainer" containerID="07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413"
Nov 02 14:09:01.546634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.546542    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:09:01.548087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.548063    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:09:01.562758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.562739    5034 scope.go:122] "RemoveContainer" containerID="dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc"
Nov 02 14:09:01.578525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.578497    5034 scope.go:122] "RemoveContainer" containerID="07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413"
Nov 02 14:09:01.578954 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:09:01.578906    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413\": container with ID starting with 07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413 not found: ID does not exist" containerID="07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413"
Nov 02 14:09:01.579069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.578953    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413"} err="failed to get container status \"07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413\": rpc error: code = NotFound desc = could not find container \"07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413\": container with ID starting with 07ad7d1d688569a63ec4e451ba5398c9b9c43baef83e3ff62c9bea5e34ba9413 not found: ID does not exist"
Nov 02 14:09:01.579069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.578986    5034 scope.go:122] "RemoveContainer" containerID="dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc"
Nov 02 14:09:01.588099 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:09:01.587873    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc\": container with ID starting with dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc not found: ID does not exist" containerID="dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc"
Nov 02 14:09:01.588099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.587918    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc"} err="failed to get container status \"dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc\": rpc error: code = NotFound desc = could not find container \"dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc\": container with ID starting with dbc15d424e9a1c55a5ff581a14001318f50a44f542b60544791f774880e7b5bc not found: ID does not exist"
Nov 02 14:09:01.664502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.664424    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"]
Nov 02 14:09:01.688893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:01.683153    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b7489f-openshift-default-54f44c5c69-ph7cd"]
Nov 02 14:09:02.554838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:02.554737    5034 generic.go:356] "Generic (PLEG): container finished" podID="5048905b-cee0-4c96-ab97-a91a67d749db" containerID="990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45" exitCode=0
Nov 02 14:09:02.554838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:02.554808    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" event={"ID":"5048905b-cee0-4c96-ab97-a91a67d749db","Type":"ContainerDied","Data":"990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45"}
Nov 02 14:09:02.555396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:02.555377    5034 scope.go:122] "RemoveContainer" containerID="990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45"
Nov 02 14:09:02.556947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:02.556914    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:02.557066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:02.556998    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:02.557463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:02.557071    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:03.401411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:03.401377    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c57591-6c30-490a-b348-e97fbbd80319" path="/var/lib/kubelet/pods/15c57591-6c30-490a-b348-e97fbbd80319/volumes"
Nov 02 14:09:03.566019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:03.565985    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" event={"ID":"5048905b-cee0-4c96-ab97-a91a67d749db","Type":"ContainerStarted","Data":"b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00"}
Nov 02 14:09:03.718010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:03.717922    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:09:03.719453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:03.719432    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 14:09:03.719571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:03.719484    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 14:09:03.919792 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:03.919754    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:09:04.035522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:04.035490    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"]
Nov 02 14:09:04.040589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:04.040566    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14749f85-tkhrm"]
Nov 02 14:09:04.717399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:04.717366    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 14:09:04.717799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:04.717420    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 14:09:05.405271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:05.405229    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf95407d-4a33-490f-9c03-6294ba0eb004" path="/var/lib/kubelet/pods/cf95407d-4a33-490f-9c03-6294ba0eb004/volumes"
Nov 02 14:09:05.717574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:05.717494    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 14:09:05.718006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:05.717568    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 14:09:06.321278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.321245    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"]
Nov 02 14:09:06.321908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.321893    5034 cpu_manager.go:414] "containerMap: removing container" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy"
Nov 02 14:09:06.321975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.321909    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae86fff-9730-4845-b60e-a0eac4259ef8" containerName="istio-proxy"
Nov 02 14:09:06.321975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.321938    5034 cpu_manager.go:414] "containerMap: removing container" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy"
Nov 02 14:09:06.321975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.321943    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy"
Nov 02 14:09:06.321975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.321953    5034 cpu_manager.go:414] "containerMap: removing container" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy"
Nov 02 14:09:06.321975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.321958    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy"
Nov 02 14:09:06.322136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.322052    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy"
Nov 02 14:09:06.322136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.322060    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c57591-6c30-490a-b348-e97fbbd80319" containerName="istio-proxy"
Nov 02 14:09:06.325252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.325227    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:06.356351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.356325    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-8acd34-dockercfg-h66w7\""
Nov 02 14:09:06.356765 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.356609    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-8acd34\""
Nov 02 14:09:06.362055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.362029    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"]
Nov 02 14:09:06.411802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.399851    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6" podUID="72d68081-4a0b-4e95-9e37-c5c8718cc589" containerName="payload-processing" containerID="cri-o://652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739" gracePeriod=25
Nov 02 14:09:06.575753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.575666    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"]
Nov 02 14:09:06.588179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.582948    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96268ea-20ee-4a56-abe0-1d20f56dcecd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q\" (UID: \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:06.588179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.583109    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmx9\" (UniqueName: \"kubernetes.io/projected/b96268ea-20ee-4a56-abe0-1d20f56dcecd-kube-api-access-tbmx9\") pod \"payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q\" (UID: \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:06.630635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.630606    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:06.647530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.647500    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"]
Nov 02 14:09:06.695803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.693136    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96268ea-20ee-4a56-abe0-1d20f56dcecd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q\" (UID: \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:06.695803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.693297    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmx9\" (UniqueName: \"kubernetes.io/projected/b96268ea-20ee-4a56-abe0-1d20f56dcecd-kube-api-access-tbmx9\") pod \"payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q\" (UID: \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:06.695803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.694933    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96268ea-20ee-4a56-abe0-1d20f56dcecd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q\" (UID: \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:06.717774 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.717735    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 14:09:06.718702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.718311    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 14:09:06.736493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.736463    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmx9\" (UniqueName: \"kubernetes.io/projected/b96268ea-20ee-4a56-abe0-1d20f56dcecd-kube-api-access-tbmx9\") pod \"payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q\" (UID: \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:06.813828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.794842    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcj28\" (UniqueName: \"kubernetes.io/projected/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-kube-api-access-hcj28\") pod \"payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm\" (UID: \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:06.813828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.794957    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm\" (UID: \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:06.905848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.903442    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcj28\" (UniqueName: \"kubernetes.io/projected/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-kube-api-access-hcj28\") pod \"payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm\" (UID: \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:06.905848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.903559    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm\" (UID: \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:06.906085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.906049    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm\" (UID: \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:06.922358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.921988    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:09:06.929589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.929561    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcj28\" (UniqueName: \"kubernetes.io/projected/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-kube-api-access-hcj28\") pod \"payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm\" (UID: \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:06.940048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.940025    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:06.965435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:06.965401    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:07.009801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.008440    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d68081-4a0b-4e95-9e37-c5c8718cc589-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d68081-4a0b-4e95-9e37-c5c8718cc589-plugins-config-volume\") pod \"72d68081-4a0b-4e95-9e37-c5c8718cc589\" (UID: \"72d68081-4a0b-4e95-9e37-c5c8718cc589\") "
Nov 02 14:09:07.009801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.008738    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d68081-4a0b-4e95-9e37-c5c8718cc589-kube-api-access-xd97n\" (UniqueName: \"kubernetes.io/projected/72d68081-4a0b-4e95-9e37-c5c8718cc589-kube-api-access-xd97n\") pod \"72d68081-4a0b-4e95-9e37-c5c8718cc589\" (UID: \"72d68081-4a0b-4e95-9e37-c5c8718cc589\") "
Nov 02 14:09:07.027447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.027411    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d68081-4a0b-4e95-9e37-c5c8718cc589-plugins-config-volume" pod "72d68081-4a0b-4e95-9e37-c5c8718cc589" (UID: "72d68081-4a0b-4e95-9e37-c5c8718cc589"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:09:07.031520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.031475    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d68081-4a0b-4e95-9e37-c5c8718cc589-kube-api-access-xd97n" pod "72d68081-4a0b-4e95-9e37-c5c8718cc589" (UID: "72d68081-4a0b-4e95-9e37-c5c8718cc589"). InnerVolumeSpecName "kube-api-access-xd97n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:09:07.144385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.139649    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd97n\" (UniqueName: \"kubernetes.io/projected/72d68081-4a0b-4e95-9e37-c5c8718cc589-kube-api-access-xd97n\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:07.144385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.139685    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d68081-4a0b-4e95-9e37-c5c8718cc589-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:09:07.322241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.322204    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"]
Nov 02 14:09:07.533450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.533417    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"]
Nov 02 14:09:07.729883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.729791    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 02 14:09:07.729883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.729848    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 02 14:09:07.750076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.749452    5034 generic.go:356] "Generic (PLEG): container finished" podID="72d68081-4a0b-4e95-9e37-c5c8718cc589" containerID="652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739" exitCode=0
Nov 02 14:09:07.750076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.749630    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6" event={"ID":"72d68081-4a0b-4e95-9e37-c5c8718cc589","Type":"ContainerDied","Data":"652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739"}
Nov 02 14:09:07.750076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.749680    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6" event={"ID":"72d68081-4a0b-4e95-9e37-c5c8718cc589","Type":"ContainerDied","Data":"2947607379dfb0b55e8edfce852fb6c2b86f51dc709d0222a5213dad002c3e60"}
Nov 02 14:09:07.750076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.749710    5034 scope.go:122] "RemoveContainer" containerID="652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739"
Nov 02 14:09:07.750076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.749870    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"
Nov 02 14:09:07.757744 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.757712    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q" event={"ID":"b96268ea-20ee-4a56-abe0-1d20f56dcecd","Type":"ContainerStarted","Data":"d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1"}
Nov 02 14:09:07.757898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.757769    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q" event={"ID":"b96268ea-20ee-4a56-abe0-1d20f56dcecd","Type":"ContainerStarted","Data":"2f555c37580cfc08402f7f23d87289a6bf5942485f1f5601065cddaada9201ec"}
Nov 02 14:09:07.758596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.758566    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:07.760507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.760478    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm" event={"ID":"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec","Type":"ContainerStarted","Data":"3282525313d3a0e5c2ab3fb9718fdd83c0103713bfa89d885578ff3de2abd65f"}
Nov 02 14:09:07.761228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.761208    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:07.819805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.811391    5034 scope.go:122] "RemoveContainer" containerID="652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739"
Nov 02 14:09:07.819805 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:09:07.812047    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739\": container with ID starting with 652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739 not found: ID does not exist" containerID="652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739"
Nov 02 14:09:07.819805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.812084    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739"} err="failed to get container status \"652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739\": rpc error: code = NotFound desc = could not find container \"652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739\": container with ID starting with 652c8fde7b7bb049fc846808128810ec31d792a524d331f8ed05cdc2d000f739 not found: ID does not exist"
Nov 02 14:09:07.968936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:07.968867    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm" podStartSLOduration=1.968841377 podStartE2EDuration="1.968841377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:07.836645781 +0000 UTC m=+1538.587950210" watchObservedRunningTime="2026-11-02 14:09:07.968841377 +0000 UTC m=+1538.720145818"
Nov 02 14:09:08.007496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:08.007436    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q" podStartSLOduration=2.007417562 podStartE2EDuration="2.007417562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:07.95941256 +0000 UTC m=+1538.710717001" watchObservedRunningTime="2026-11-02 14:09:08.007417562 +0000 UTC m=+1538.758722001"
Nov 02 14:09:08.053220 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:08.053179    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"]
Nov 02 14:09:08.062145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:08.061817    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-7594649ddc-2pdp6"]
Nov 02 14:09:08.773244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:08.773200    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm" event={"ID":"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec","Type":"ContainerStarted","Data":"ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e"}
Nov 02 14:09:09.021014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:09.020975    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:09:09.023223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:09.023194    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:09:09.396493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:09.396461    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d68081-4a0b-4e95-9e37-c5c8718cc589" path="/var/lib/kubelet/pods/72d68081-4a0b-4e95-9e37-c5c8718cc589/volumes"
Nov 02 14:09:13.509349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.509312    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"]
Nov 02 14:09:13.510328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.510300    5034 cpu_manager.go:414] "containerMap: removing container" podUID="72d68081-4a0b-4e95-9e37-c5c8718cc589" containerName="payload-processing"
Nov 02 14:09:13.510328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.510324    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d68081-4a0b-4e95-9e37-c5c8718cc589" containerName="payload-processing"
Nov 02 14:09:13.510564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.510550    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d68081-4a0b-4e95-9e37-c5c8718cc589" containerName="payload-processing"
Nov 02 14:09:13.514326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.514300    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:13.533004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.532981    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c1d69c\""
Nov 02 14:09:13.545249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.545230    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c1d69c-dockercfg-hrwnm\""
Nov 02 14:09:13.566294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.566269    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"]
Nov 02 14:09:13.594723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.594691    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t\" (UID: \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:13.594998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.594969    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgljc\" (UniqueName: \"kubernetes.io/projected/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-kube-api-access-wgljc\") pod \"payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t\" (UID: \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:13.666801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.665933    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"]
Nov 02 14:09:13.695806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.695153    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:13.750541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.737571    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"]
Nov 02 14:09:13.750541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.740086    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnzss\" (UniqueName: \"kubernetes.io/projected/f9456c6a-5a91-43ac-b70e-74b1b540f800-kube-api-access-vnzss\") pod \"payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb\" (UID: \"f9456c6a-5a91-43ac-b70e-74b1b540f800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:13.750541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.740201    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9456c6a-5a91-43ac-b70e-74b1b540f800-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb\" (UID: \"f9456c6a-5a91-43ac-b70e-74b1b540f800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:13.750541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.740298    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t\" (UID: \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:13.750541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.740452    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgljc\" (UniqueName: \"kubernetes.io/projected/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-kube-api-access-wgljc\") pod \"payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t\" (UID: \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:13.750541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.741806    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t\" (UID: \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:13.826610 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.807889    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgljc\" (UniqueName: \"kubernetes.io/projected/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-kube-api-access-wgljc\") pod \"payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t\" (UID: \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:13.829334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.827513    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:13.861846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.847486    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnzss\" (UniqueName: \"kubernetes.io/projected/f9456c6a-5a91-43ac-b70e-74b1b540f800-kube-api-access-vnzss\") pod \"payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb\" (UID: \"f9456c6a-5a91-43ac-b70e-74b1b540f800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:13.861846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.847669    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9456c6a-5a91-43ac-b70e-74b1b540f800-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb\" (UID: \"f9456c6a-5a91-43ac-b70e-74b1b540f800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:13.861846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.849071    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9456c6a-5a91-43ac-b70e-74b1b540f800-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb\" (UID: \"f9456c6a-5a91-43ac-b70e-74b1b540f800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:13.892828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:13.890547    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnzss\" (UniqueName: \"kubernetes.io/projected/f9456c6a-5a91-43ac-b70e-74b1b540f800-kube-api-access-vnzss\") pod \"payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb\" (UID: \"f9456c6a-5a91-43ac-b70e-74b1b540f800\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:14.079821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.079582    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:14.165587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.163948    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"]
Nov 02 14:09:14.167968 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:09:14.167940    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf0da69_038d_4fc3_9e8a_2a8501ff7fe7.slice/crio-06de72dd094eb11d018b5840468843fdf885f979d40c80629e5bef579cfe77cc WatchSource:0}: Error finding container 06de72dd094eb11d018b5840468843fdf885f979d40c80629e5bef579cfe77cc: Status 404 returned error can't find the container with id 06de72dd094eb11d018b5840468843fdf885f979d40c80629e5bef579cfe77cc
Nov 02 14:09:14.384393 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:09:14.384362    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9456c6a_5a91_43ac_b70e_74b1b540f800.slice/crio-c56618bcb0653a530392205f292909deb59e099f62d2c2d98a520644a4f8e5d0 WatchSource:0}: Error finding container c56618bcb0653a530392205f292909deb59e099f62d2c2d98a520644a4f8e5d0: Status 404 returned error can't find the container with id c56618bcb0653a530392205f292909deb59e099f62d2c2d98a520644a4f8e5d0
Nov 02 14:09:14.385447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.385424    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"]
Nov 02 14:09:14.845711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.845670    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb" event={"ID":"f9456c6a-5a91-43ac-b70e-74b1b540f800","Type":"ContainerStarted","Data":"d50a4c9de70682d16f92d4cd457035082ead03d0c550a61dde6702b91df8f464"}
Nov 02 14:09:14.846167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.845743    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:14.846167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.845771    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb" event={"ID":"f9456c6a-5a91-43ac-b70e-74b1b540f800","Type":"ContainerStarted","Data":"c56618bcb0653a530392205f292909deb59e099f62d2c2d98a520644a4f8e5d0"}
Nov 02 14:09:14.847687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.847662    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t" event={"ID":"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7","Type":"ContainerStarted","Data":"de42bd82d3b29289ec0dbcb8b1a6d1e73864959ce1be579ead460ff58782773c"}
Nov 02 14:09:14.847855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.847700    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t" event={"ID":"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7","Type":"ContainerStarted","Data":"06de72dd094eb11d018b5840468843fdf885f979d40c80629e5bef579cfe77cc"}
Nov 02 14:09:14.847855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.847742    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:14.890333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.890290    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb" podStartSLOduration=1.890275109 podStartE2EDuration="1.890275109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:14.888228643 +0000 UTC m=+1545.639533082" watchObservedRunningTime="2026-11-02 14:09:14.890275109 +0000 UTC m=+1545.641579567"
Nov 02 14:09:14.931859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:14.931812    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t" podStartSLOduration=1.9317674519999999 podStartE2EDuration="1.931767452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:14.931314902 +0000 UTC m=+1545.682619340" watchObservedRunningTime="2026-11-02 14:09:14.931767452 +0000 UTC m=+1545.683071892"
Nov 02 14:09:18.415517 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:09:18.415474    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:09:19.825807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:19.820101    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:09:19.825807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:19.821668    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:09:20.073920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.073887    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"]
Nov 02 14:09:20.111077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.110985    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"]
Nov 02 14:09:20.114692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.111827    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:20.138525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.138498    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-13367a-dockercfg-gcbf6\""
Nov 02 14:09:20.138705 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.138651    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-13367a\""
Nov 02 14:09:20.219823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.219767    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x\" (UID: \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:20.220064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.220041    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbbwp\" (UniqueName: \"kubernetes.io/projected/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-kube-api-access-wbbwp\") pod \"payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x\" (UID: \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:20.321396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.321357    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbbwp\" (UniqueName: \"kubernetes.io/projected/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-kube-api-access-wbbwp\") pod \"payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x\" (UID: \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:20.321562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.321474    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x\" (UID: \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:20.322246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.322225    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x\" (UID: \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:20.345828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.345802    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbbwp\" (UniqueName: \"kubernetes.io/projected/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-kube-api-access-wbbwp\") pod \"payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x\" (UID: \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:20.447332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.447254    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:20.802192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.802158    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"]
Nov 02 14:09:20.811522 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:09:20.811482    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80a5c73_5ac2_45cb_9a76_ce646d16fb2b.slice/crio-0b08e3a11da87d7bfb0bb71321b1fbdd830695aacaaf2817322dca2cbc3208a1 WatchSource:0}: Error finding container 0b08e3a11da87d7bfb0bb71321b1fbdd830695aacaaf2817322dca2cbc3208a1: Status 404 returned error can't find the container with id 0b08e3a11da87d7bfb0bb71321b1fbdd830695aacaaf2817322dca2cbc3208a1
Nov 02 14:09:20.910244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.910209    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x" event={"ID":"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b","Type":"ContainerStarted","Data":"0b08e3a11da87d7bfb0bb71321b1fbdd830695aacaaf2817322dca2cbc3208a1"}
Nov 02 14:09:20.911353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.911330    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:20.955727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:20.955671    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x" podStartSLOduration=0.955655913 podStartE2EDuration="955.655913ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:20.95368625 +0000 UTC m=+1551.704990688" watchObservedRunningTime="2026-11-02 14:09:20.955655913 +0000 UTC m=+1551.706960350"
Nov 02 14:09:21.921545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:21.921514    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x" event={"ID":"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b","Type":"ContainerStarted","Data":"5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421"}
Nov 02 14:09:25.858176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:25.858143    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:09:25.858568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:25.858219    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:09:32.938394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:32.938363    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:09:35.524683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:35.524654    5034 scope.go:122] "RemoveContainer" containerID="cff1ad560af18b84e9337c4985a155d1a0648a378a54aa10006cc6b67143f8d0"
Nov 02 14:09:38.014132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.014103    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"]
Nov 02 14:09:38.020474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.020441    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.037319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.037291    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-370905-openshift-default-dockercfg-gkdjd\""
Nov 02 14:09:38.057660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.057625    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"]
Nov 02 14:09:38.082460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.082413    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2fsr\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-kube-api-access-n2fsr\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.082739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.082721    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-credential-socket\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.082963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.082945    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f666d7f-79d0-4556-b859-62fd24beb2a0-istiod-ca-cert\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.083597 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.083575    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-data\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.083714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.083659    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-token\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.083950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.083928    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-certs\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.084182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.084162    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-socket\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.084349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.084333    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-podinfo\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.084501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.084486    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-envoy\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.260826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.241445    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-certs\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.272848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.261126    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-socket\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.273123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.273096    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-podinfo\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.273724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.273701    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-envoy\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.274168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.274121    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2fsr\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-kube-api-access-n2fsr\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.274351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.274329    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-credential-socket\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.274502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.274481    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f666d7f-79d0-4556-b859-62fd24beb2a0-istiod-ca-cert\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.274737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.274720    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-data\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.274949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.274932    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-token\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.283802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.272189    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-socket\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.286891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.243131    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-certs\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.286891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.284585    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f666d7f-79d0-4556-b859-62fd24beb2a0-istiod-ca-cert\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.286891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.284819    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-credential-socket\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.286891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.285039    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-data\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.287254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.287232    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-envoy\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.297498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.297477    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-podinfo\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.307797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.301683    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-token\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.315474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.315416    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2fsr\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-kube-api-access-n2fsr\") pod \"e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") " pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.359524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.359497    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:38.685274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.685146    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"]
Nov 02 14:09:38.685413 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:09:38.685318    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f666d7f_79d0_4556_b859_62fd24beb2a0.slice/crio-b0e7c0e4074d2855cd6d7fcd8e730c43c2ecc4b6c92bd3a32be78bfd31ab56de WatchSource:0}: Error finding container b0e7c0e4074d2855cd6d7fcd8e730c43c2ecc4b6c92bd3a32be78bfd31ab56de: Status 404 returned error can't find the container with id b0e7c0e4074d2855cd6d7fcd8e730c43c2ecc4b6c92bd3a32be78bfd31ab56de
Nov 02 14:09:38.688759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.688722    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:38.688871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.688827    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:38.688916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:38.688878    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:39.103247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:39.103202    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" event={"ID":"8f666d7f-79d0-4556-b859-62fd24beb2a0","Type":"ContainerStarted","Data":"d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328"}
Nov 02 14:09:39.103728 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:39.103276    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" event={"ID":"8f666d7f-79d0-4556-b859-62fd24beb2a0","Type":"ContainerStarted","Data":"b0e7c0e4074d2855cd6d7fcd8e730c43c2ecc4b6c92bd3a32be78bfd31ab56de"}
Nov 02 14:09:39.173109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:39.169807    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" podStartSLOduration=2.169773233 podStartE2EDuration="2.169773233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:39.165479945 +0000 UTC m=+1569.916784387" watchObservedRunningTime="2026-11-02 14:09:39.169773233 +0000 UTC m=+1569.921077673"
Nov 02 14:09:39.365183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:39.365022    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:39.381807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:39.376341    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:09:39.381807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:39.376400    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 14:09:40.366115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:40.366033    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:41.120136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:41.120098    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:41.121107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:41.121088    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:09:43.652167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.652131    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"]
Nov 02 14:09:43.657563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.657540    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:43.676799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.675144    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-370905-dockercfg-kdv7r\""
Nov 02 14:09:43.703297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.703253    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-370905\""
Nov 02 14:09:43.709520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.709491    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"]
Nov 02 14:09:43.811825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.798872    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f30da4-caef-4251-bcd1-9c242bef4d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r\" (UID: \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:43.811825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.798981    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvns\" (UniqueName: \"kubernetes.io/projected/f4f30da4-caef-4251-bcd1-9c242bef4d4b-kube-api-access-9zvns\") pod \"payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r\" (UID: \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:43.914578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.904506    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f30da4-caef-4251-bcd1-9c242bef4d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r\" (UID: \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:43.914578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.904618    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvns\" (UniqueName: \"kubernetes.io/projected/f4f30da4-caef-4251-bcd1-9c242bef4d4b-kube-api-access-9zvns\") pod \"payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r\" (UID: \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:43.914578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.914425    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f30da4-caef-4251-bcd1-9c242bef4d4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r\" (UID: \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:43.965296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.965248    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvns\" (UniqueName: \"kubernetes.io/projected/f4f30da4-caef-4251-bcd1-9c242bef4d4b-kube-api-access-9zvns\") pod \"payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r\" (UID: \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:44.000801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:43.985490    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:44.400311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:44.400282    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"]
Nov 02 14:09:44.405041 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:09:44.405017    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f30da4_caef_4251_bcd1_9c242bef4d4b.slice/crio-069f74a1167092986c64e122e32585078fd3bd3898a359047bdad8e15c3dd98c WatchSource:0}: Error finding container 069f74a1167092986c64e122e32585078fd3bd3898a359047bdad8e15c3dd98c: Status 404 returned error can't find the container with id 069f74a1167092986c64e122e32585078fd3bd3898a359047bdad8e15c3dd98c
Nov 02 14:09:45.156159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:45.156125    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r" event={"ID":"f4f30da4-caef-4251-bcd1-9c242bef4d4b","Type":"ContainerStarted","Data":"ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a"}
Nov 02 14:09:45.156566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:45.156178    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r" event={"ID":"f4f30da4-caef-4251-bcd1-9c242bef4d4b","Type":"ContainerStarted","Data":"069f74a1167092986c64e122e32585078fd3bd3898a359047bdad8e15c3dd98c"}
Nov 02 14:09:45.156566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:45.156292    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:45.212955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:45.212896    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r" podStartSLOduration=2.21287759 podStartE2EDuration="2.21287759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:45.212598586 +0000 UTC m=+1575.963903024" watchObservedRunningTime="2026-11-02 14:09:45.21287759 +0000 UTC m=+1575.964182031"
Nov 02 14:09:47.548469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:47.548426    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:09:47.548965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:47.548498    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:09:48.417353 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:09:48.417301    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:09:50.211573 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:50.211500    5034 generic.go:356] "Generic (PLEG): container finished" podID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerID="9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308" exitCode=0
Nov 02 14:09:50.213894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:50.213582    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" event={"ID":"60cfd239-3bd1-468f-81e1-bb85fee11656","Type":"ContainerDied","Data":"9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308"}
Nov 02 14:09:50.214702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:50.214672    5034 scope.go:122] "RemoveContainer" containerID="9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308"
Nov 02 14:09:50.219218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:50.219186    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:50.219330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:50.219292    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:50.219330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:50.219326    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:51.224399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:51.224365    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" event={"ID":"60cfd239-3bd1-468f-81e1-bb85fee11656","Type":"ContainerStarted","Data":"ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2"}
Nov 02 14:09:51.434891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:51.434856    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:09:51.435134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:51.435112    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:09:51.435183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:51.435161    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 14:09:52.435354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:52.435315    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:09:52.435893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:52.435380    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 14:09:53.435340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:53.435305    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:09:53.435522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:53.435366    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 14:09:54.435246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:54.435215    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:09:54.435433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:54.435303    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 14:09:55.344511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.344475    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"]
Nov 02 14:09:55.352549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.352526    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.410830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.400266    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-732d3a-openshift-default-dockercfg-8q7wz\""
Nov 02 14:09:55.425179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.425153    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"]
Nov 02 14:09:55.435869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.435840    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:09:55.436013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.435896    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 14:09:55.645863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.645684    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-socket\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.646048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.645962    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-credential-socket\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.646113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.646099    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-token\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.646167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.646152    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-envoy\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.646224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.646214    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-podinfo\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.657801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.646291    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-certs\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.657801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.646326    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c3817b-5573-4131-8ec9-59a101e298f8-istiod-ca-cert\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.657801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.646420    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-data\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.657801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.646451    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pz9c\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-kube-api-access-7pz9c\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.747602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.747561    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-token\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.747830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.747616    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-envoy\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.747830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.747660    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-podinfo\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.747830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.747719    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-certs\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.747830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.747746    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c3817b-5573-4131-8ec9-59a101e298f8-istiod-ca-cert\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.748076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.747848    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-data\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.748076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.747875    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pz9c\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-kube-api-access-7pz9c\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.748076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.747958    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-socket\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.748231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.748115    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-credential-socket\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.748291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.748272    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-data\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.748527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.748478    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-socket\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.748669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.748627    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-credential-socket\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.748747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.748663    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-certs\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.748747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.748661    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c3817b-5573-4131-8ec9-59a101e298f8-istiod-ca-cert\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.750077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.750057    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-envoy\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.750406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.750387    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-podinfo\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.781682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.781628    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pz9c\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-kube-api-access-7pz9c\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:55.785296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:55.785268    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-token\") pod \"e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") " pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:56.022026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:56.021973    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:56.168363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:56.168335    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:09:56.393612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:56.393581    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"]
Nov 02 14:09:56.395218 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:09:56.395180    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c3817b_5573_4131_8ec9_59a101e298f8.slice/crio-a16a6ab70b1496dc2b1dbec8878728c3b3d307df54f328ff8e9b7f5e3c3eb16a WatchSource:0}: Error finding container a16a6ab70b1496dc2b1dbec8878728c3b3d307df54f328ff8e9b7f5e3c3eb16a: Status 404 returned error can't find the container with id a16a6ab70b1496dc2b1dbec8878728c3b3d307df54f328ff8e9b7f5e3c3eb16a
Nov 02 14:09:56.397958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:56.397926    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:56.398085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:56.398006    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:56.398085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:56.398054    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:09:56.435497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:56.435471    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Nov 02 14:09:56.435619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:56.435523    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Nov 02 14:09:57.272513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:57.272478    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" event={"ID":"a7c3817b-5573-4131-8ec9-59a101e298f8","Type":"ContainerStarted","Data":"89990b0326ad6f86c2b4553aa75ae441ff81d5ef643f1e87b69ba8b9ef3c48d4"}
Nov 02 14:09:57.272689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:57.272529    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" event={"ID":"a7c3817b-5573-4131-8ec9-59a101e298f8","Type":"ContainerStarted","Data":"a16a6ab70b1496dc2b1dbec8878728c3b3d307df54f328ff8e9b7f5e3c3eb16a"}
Nov 02 14:09:57.336834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:57.336764    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" podStartSLOduration=2.336744784 podStartE2EDuration="2.336744784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:09:57.331405902 +0000 UTC m=+1588.082710364" watchObservedRunningTime="2026-11-02 14:09:57.336744784 +0000 UTC m=+1588.088049223"
Nov 02 14:09:57.439019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:57.438992    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:09:57.439410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:57.439393    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:09:57.440181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:57.440158    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:09:58.022419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:58.022389    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:58.029067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:58.029037    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:58.281497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:58.281402    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:09:58.282411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:09:58.282386    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:10:01.667121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.667088    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"]
Nov 02 14:10:01.671789 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.671755    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:01.701010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.700982    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"]
Nov 02 14:10:01.708362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.708336    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-732d3a\""
Nov 02 14:10:01.708513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.708475    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-732d3a-dockercfg-g4xrl\""
Nov 02 14:10:01.744869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.744825    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxqzb\" (UniqueName: \"kubernetes.io/projected/8d0062d3-bf93-44ad-a827-85760d9cb85e-kube-api-access-qxqzb\") pod \"payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj\" (UID: \"8d0062d3-bf93-44ad-a827-85760d9cb85e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:01.745171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.744989    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0062d3-bf93-44ad-a827-85760d9cb85e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj\" (UID: \"8d0062d3-bf93-44ad-a827-85760d9cb85e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:01.887800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.887697    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0062d3-bf93-44ad-a827-85760d9cb85e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj\" (UID: \"8d0062d3-bf93-44ad-a827-85760d9cb85e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:01.896801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.888003    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxqzb\" (UniqueName: \"kubernetes.io/projected/8d0062d3-bf93-44ad-a827-85760d9cb85e-kube-api-access-qxqzb\") pod \"payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj\" (UID: \"8d0062d3-bf93-44ad-a827-85760d9cb85e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:01.916564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.913417    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0062d3-bf93-44ad-a827-85760d9cb85e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj\" (UID: \"8d0062d3-bf93-44ad-a827-85760d9cb85e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:01.949134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.946913    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxqzb\" (UniqueName: \"kubernetes.io/projected/8d0062d3-bf93-44ad-a827-85760d9cb85e-kube-api-access-qxqzb\") pod \"payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj\" (UID: \"8d0062d3-bf93-44ad-a827-85760d9cb85e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:01.988532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:01.988491    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:02.294679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:02.294647    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"]
Nov 02 14:10:02.295252 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:10:02.295226    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0062d3_bf93_44ad_a827_85760d9cb85e.slice/crio-d8803f9c68004dd236f02d4c30f4cd858e1289569f7204892e3aca43f29c296b WatchSource:0}: Error finding container d8803f9c68004dd236f02d4c30f4cd858e1289569f7204892e3aca43f29c296b: Status 404 returned error can't find the container with id d8803f9c68004dd236f02d4c30f4cd858e1289569f7204892e3aca43f29c296b
Nov 02 14:10:02.393732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:02.393689    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj" event={"ID":"8d0062d3-bf93-44ad-a827-85760d9cb85e","Type":"ContainerStarted","Data":"d8803f9c68004dd236f02d4c30f4cd858e1289569f7204892e3aca43f29c296b"}
Nov 02 14:10:03.405291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:03.405257    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj" event={"ID":"8d0062d3-bf93-44ad-a827-85760d9cb85e","Type":"ContainerStarted","Data":"7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf"}
Nov 02 14:10:03.405697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:03.405317    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:03.449844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:03.449788    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj" podStartSLOduration=2.449763017 podStartE2EDuration="2.449763017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:10:03.44802493 +0000 UTC m=+1594.199329368" watchObservedRunningTime="2026-11-02 14:10:03.449763017 +0000 UTC m=+1594.201067454"
Nov 02 14:10:14.415603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:14.415574    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:10:18.380690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:18.380654    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"]
Nov 02 14:10:18.415970 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:10:18.415931    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:10:18.427346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:18.427319    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"]
Nov 02 14:10:23.381920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.381439    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv" podUID="d13007cc-d6c8-447a-9653-d40f74514064" containerName="payload-processing" containerID="cri-o://629db5881abf8cebd6c393fd349e55218bfd80a3810f530110e95834bbfe4dfb" gracePeriod=25
Nov 02 14:10:23.428322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.428286    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2" podUID="ac4633fd-49f3-4cf4-8457-c89cc30aa4dd" containerName="payload-pre-processing" containerID="cri-o://e8eb0a3c04fb4c7ec6d33926d1dea3a1c971427d4e89ad95842b231a1c5f6b79" gracePeriod=25
Nov 02 14:10:23.729926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.721750    5034 generic.go:356] "Generic (PLEG): container finished" podID="d13007cc-d6c8-447a-9653-d40f74514064" containerID="629db5881abf8cebd6c393fd349e55218bfd80a3810f530110e95834bbfe4dfb" exitCode=0
Nov 02 14:10:23.729926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.722030    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv" event={"ID":"d13007cc-d6c8-447a-9653-d40f74514064","Type":"ContainerDied","Data":"629db5881abf8cebd6c393fd349e55218bfd80a3810f530110e95834bbfe4dfb"}
Nov 02 14:10:23.729926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.724208    5034 generic.go:356] "Generic (PLEG): container finished" podID="ac4633fd-49f3-4cf4-8457-c89cc30aa4dd" containerID="e8eb0a3c04fb4c7ec6d33926d1dea3a1c971427d4e89ad95842b231a1c5f6b79" exitCode=0
Nov 02 14:10:23.729926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.724337    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2" event={"ID":"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd","Type":"ContainerDied","Data":"e8eb0a3c04fb4c7ec6d33926d1dea3a1c971427d4e89ad95842b231a1c5f6b79"}
Nov 02 14:10:23.909573 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.909547    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:10:23.932903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.932879    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:10:23.972951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.972922    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13007cc-d6c8-447a-9653-d40f74514064-kube-api-access-wk2qp\" (UniqueName: \"kubernetes.io/projected/d13007cc-d6c8-447a-9653-d40f74514064-kube-api-access-wk2qp\") pod \"d13007cc-d6c8-447a-9653-d40f74514064\" (UID: \"d13007cc-d6c8-447a-9653-d40f74514064\") "
Nov 02 14:10:23.973116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.973014    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-plugins-config-volume\") pod \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\" (UID: \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\") "
Nov 02 14:10:23.973116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.973108    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-kube-api-access-rq7pz\" (UniqueName: \"kubernetes.io/projected/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-kube-api-access-rq7pz\") pod \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\" (UID: \"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd\") "
Nov 02 14:10:23.973236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.973198    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13007cc-d6c8-447a-9653-d40f74514064-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13007cc-d6c8-447a-9653-d40f74514064-plugins-config-volume\") pod \"d13007cc-d6c8-447a-9653-d40f74514064\" (UID: \"d13007cc-d6c8-447a-9653-d40f74514064\") "
Nov 02 14:10:23.973768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.973622    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13007cc-d6c8-447a-9653-d40f74514064-plugins-config-volume" pod "d13007cc-d6c8-447a-9653-d40f74514064" (UID: "d13007cc-d6c8-447a-9653-d40f74514064"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:23.973935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.973817    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-plugins-config-volume" pod "ac4633fd-49f3-4cf4-8457-c89cc30aa4dd" (UID: "ac4633fd-49f3-4cf4-8457-c89cc30aa4dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:23.975481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.975458    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13007cc-d6c8-447a-9653-d40f74514064-kube-api-access-wk2qp" pod "d13007cc-d6c8-447a-9653-d40f74514064" (UID: "d13007cc-d6c8-447a-9653-d40f74514064"). InnerVolumeSpecName "kube-api-access-wk2qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:23.975972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:23.975949    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-kube-api-access-rq7pz" pod "ac4633fd-49f3-4cf4-8457-c89cc30aa4dd" (UID: "ac4633fd-49f3-4cf4-8457-c89cc30aa4dd"). InnerVolumeSpecName "kube-api-access-rq7pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:24.075423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.075390    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk2qp\" (UniqueName: \"kubernetes.io/projected/d13007cc-d6c8-447a-9653-d40f74514064-kube-api-access-wk2qp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:24.075423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.075419    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:24.075637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.075433    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq7pz\" (UniqueName: \"kubernetes.io/projected/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd-kube-api-access-rq7pz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:24.075637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.075446    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d13007cc-d6c8-447a-9653-d40f74514064-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:24.733163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.733136    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"
Nov 02 14:10:24.733625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.733173    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2" event={"ID":"ac4633fd-49f3-4cf4-8457-c89cc30aa4dd","Type":"ContainerDied","Data":"c71d440bd3620f89a228b936658de2b902c0787e04907dad60eeafb8587effd1"}
Nov 02 14:10:24.733625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.733246    5034 scope.go:122] "RemoveContainer" containerID="e8eb0a3c04fb4c7ec6d33926d1dea3a1c971427d4e89ad95842b231a1c5f6b79"
Nov 02 14:10:24.735071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.735044    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"
Nov 02 14:10:24.735188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.735072    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv" event={"ID":"d13007cc-d6c8-447a-9653-d40f74514064","Type":"ContainerDied","Data":"46962d4dc9055c2cef9309b8a2315d01a70373b0c81d4196710d83ba4df15d6c"}
Nov 02 14:10:24.747964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.747948    5034 scope.go:122] "RemoveContainer" containerID="629db5881abf8cebd6c393fd349e55218bfd80a3810f530110e95834bbfe4dfb"
Nov 02 14:10:24.816342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.816272    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"]
Nov 02 14:10:24.825185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.825159    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b7d400-776576f765-94mvv"]
Nov 02 14:10:24.871303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.871271    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"]
Nov 02 14:10:24.880115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:24.880086    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b7d400-7b776f4d99-s5kd2"]
Nov 02 14:10:25.397246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:25.397207    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4633fd-49f3-4cf4-8457-c89cc30aa4dd" path="/var/lib/kubelet/pods/ac4633fd-49f3-4cf4-8457-c89cc30aa4dd/volumes"
Nov 02 14:10:25.397850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:25.397831    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13007cc-d6c8-447a-9653-d40f74514064" path="/var/lib/kubelet/pods/d13007cc-d6c8-447a-9653-d40f74514064/volumes"
Nov 02 14:10:27.146348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:27.146314    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"]
Nov 02 14:10:27.146921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:27.146516    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy" containerID="cri-o://d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff" gracePeriod=30
Nov 02 14:10:28.454506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:28.454471    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:10:28.454961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:28.454531    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:10:30.934495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:30.934464    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"]
Nov 02 14:10:30.934918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:30.934687    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy" containerID="cri-o://08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf" gracePeriod=30
Nov 02 14:10:32.437096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.437028    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 14:10:32.508338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508310    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-credential-socket\") pod \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") "
Nov 02 14:10:32.508491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508380    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-certs\") pod \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") "
Nov 02 14:10:32.508491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508470    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istiod-ca-cert\") pod \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") "
Nov 02 14:10:32.508587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508514    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-socket\") pod \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") "
Nov 02 14:10:32.508653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508592    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-podinfo\") pod \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") "
Nov 02 14:10:32.508653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508609    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-credential-socket" pod "8b9df9c2-2c16-4bb5-bb05-ee1667367dba" (UID: "8b9df9c2-2c16-4bb5-bb05-ee1667367dba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:32.508653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508627    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-envoy\") pod \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") "
Nov 02 14:10:32.508856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508670    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-kube-api-access-c9k2f\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-kube-api-access-c9k2f\") pod \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") "
Nov 02 14:10:32.508856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508713    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-certs" pod "8b9df9c2-2c16-4bb5-bb05-ee1667367dba" (UID: "8b9df9c2-2c16-4bb5-bb05-ee1667367dba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:32.508856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508762    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-data\") pod \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") "
Nov 02 14:10:32.509026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508875    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-token\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-token\") pod \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\" (UID: \"8b9df9c2-2c16-4bb5-bb05-ee1667367dba\") "
Nov 02 14:10:32.509026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508889    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-socket" pod "8b9df9c2-2c16-4bb5-bb05-ee1667367dba" (UID: "8b9df9c2-2c16-4bb5-bb05-ee1667367dba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:32.509026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.508899    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istiod-ca-cert" pod "8b9df9c2-2c16-4bb5-bb05-ee1667367dba" (UID: "8b9df9c2-2c16-4bb5-bb05-ee1667367dba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:32.509240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.509215    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-data" pod "8b9df9c2-2c16-4bb5-bb05-ee1667367dba" (UID: "8b9df9c2-2c16-4bb5-bb05-ee1667367dba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:32.509750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.509721    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:32.509882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.509749    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:32.509882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.509768    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:32.509882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.509802    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:32.509882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.509815    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:32.511078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.511055    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-envoy" pod "8b9df9c2-2c16-4bb5-bb05-ee1667367dba" (UID: "8b9df9c2-2c16-4bb5-bb05-ee1667367dba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:32.511200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.511186    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-podinfo" pod "8b9df9c2-2c16-4bb5-bb05-ee1667367dba" (UID: "8b9df9c2-2c16-4bb5-bb05-ee1667367dba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:10:32.511495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.511474    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-token" pod "8b9df9c2-2c16-4bb5-bb05-ee1667367dba" (UID: "8b9df9c2-2c16-4bb5-bb05-ee1667367dba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:32.511694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.511673    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-kube-api-access-c9k2f" pod "8b9df9c2-2c16-4bb5-bb05-ee1667367dba" (UID: "8b9df9c2-2c16-4bb5-bb05-ee1667367dba"). InnerVolumeSpecName "kube-api-access-c9k2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:32.611331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.611304    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:32.611331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.611329    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:32.611331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.611337    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9k2f\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-kube-api-access-c9k2f\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:32.611662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.611347    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b9df9c2-2c16-4bb5-bb05-ee1667367dba-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:32.800129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.800096    5034 generic.go:356] "Generic (PLEG): container finished" podID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerID="d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff" exitCode=0
Nov 02 14:10:32.800324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.800143    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"
Nov 02 14:10:32.800324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.800182    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" event={"ID":"8b9df9c2-2c16-4bb5-bb05-ee1667367dba","Type":"ContainerDied","Data":"d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff"}
Nov 02 14:10:32.800324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.800248    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6" event={"ID":"8b9df9c2-2c16-4bb5-bb05-ee1667367dba","Type":"ContainerDied","Data":"4a1654168f433f142ad1376a24753458a70514b17c73cbb3c8ff4e15c54b8d9c"}
Nov 02 14:10:32.800324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.800279    5034 scope.go:122] "RemoveContainer" containerID="d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff"
Nov 02 14:10:32.811441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.811425    5034 scope.go:122] "RemoveContainer" containerID="f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df"
Nov 02 14:10:32.821176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.821154    5034 scope.go:122] "RemoveContainer" containerID="d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff"
Nov 02 14:10:32.821465 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:10:32.821448    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff\": container with ID starting with d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff not found: ID does not exist" containerID="d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff"
Nov 02 14:10:32.821531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.821472    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff"} err="failed to get container status \"d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff\": rpc error: code = NotFound desc = could not find container \"d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff\": container with ID starting with d3265fda311541e2e2eb3a7ffcf53a623557c8a06fcfb12fcbcd454905e9e5ff not found: ID does not exist"
Nov 02 14:10:32.821531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.821489    5034 scope.go:122] "RemoveContainer" containerID="f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df"
Nov 02 14:10:32.821809 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:10:32.821757    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df\": container with ID starting with f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df not found: ID does not exist" containerID="f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df"
Nov 02 14:10:32.821873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.821832    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df"} err="failed to get container status \"f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df\": rpc error: code = NotFound desc = could not find container \"f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df\": container with ID starting with f263b38bb0f4a1904e8a61df6b5f771a6320c7bbf510f7c957842cf2a38d06df not found: ID does not exist"
Nov 02 14:10:32.861365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.861343    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"]
Nov 02 14:10:32.874353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:32.874328    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-qrrm6"]
Nov 02 14:10:33.397180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:33.397140    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" path="/var/lib/kubelet/pods/8b9df9c2-2c16-4bb5-bb05-ee1667367dba/volumes"
Nov 02 14:10:35.574600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:35.574564    5034 scope.go:122] "RemoveContainer" containerID="07dd1faf6dd292d0bbe39428cdf6f0d4be20cf00927434c5cbbd2d51a0745651"
Nov 02 14:10:36.233155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.233128    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:10:36.361640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.361609    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-credential-socket\") pod \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") "
Nov 02 14:10:36.361874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.361701    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-data\") pod \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") "
Nov 02 14:10:36.361874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.361868    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-token\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-token\") pod \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") "
Nov 02 14:10:36.362017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.361879    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-credential-socket" pod "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" (UID: "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:36.362017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.361897    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-podinfo\") pod \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") "
Nov 02 14:10:36.362017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.361929    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-kube-api-access-plvxf\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-kube-api-access-plvxf\") pod \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") "
Nov 02 14:10:36.362017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.361963    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-envoy\") pod \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") "
Nov 02 14:10:36.362017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362004    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istiod-ca-cert\") pod \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") "
Nov 02 14:10:36.362262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362020    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-data" pod "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" (UID: "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:36.362262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362177    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-certs\") pod \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") "
Nov 02 14:10:36.362262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362240    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-socket\") pod \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\" (UID: \"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3\") "
Nov 02 14:10:36.362495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362468    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-certs" pod "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" (UID: "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:36.362495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362486    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istiod-ca-cert" pod "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" (UID: "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:36.362669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362650    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-socket" pod "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" (UID: "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:36.362879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362858    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.362971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362882    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.362971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362894    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.362971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362906    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.362971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.362920    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.364239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.364211    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-token" pod "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" (UID: "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:36.364457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.364441    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-podinfo" pod "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" (UID: "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:10:36.364631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.364610    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-envoy" pod "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" (UID: "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:36.364631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.364620    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-kube-api-access-plvxf" pod "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" (UID: "7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3"). InnerVolumeSpecName "kube-api-access-plvxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:36.465191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.465120    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.465191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.465148    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.465191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.465166    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.465191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.465186    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plvxf\" (UniqueName: \"kubernetes.io/projected/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3-kube-api-access-plvxf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:36.836240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.836203    5034 generic.go:356] "Generic (PLEG): container finished" podID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerID="08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf" exitCode=0
Nov 02 14:10:36.836712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.836257    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"
Nov 02 14:10:36.836712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.836291    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" event={"ID":"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3","Type":"ContainerDied","Data":"08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf"}
Nov 02 14:10:36.836712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.836393    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8" event={"ID":"7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3","Type":"ContainerDied","Data":"37a1fd5a0e198843d1b13ba25be31deb1ca521ef5119ab70bc7c827a53401a8b"}
Nov 02 14:10:36.836712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.836423    5034 scope.go:122] "RemoveContainer" containerID="08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf"
Nov 02 14:10:36.852136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.852118    5034 scope.go:122] "RemoveContainer" containerID="08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf"
Nov 02 14:10:36.852435 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:10:36.852417    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf\": container with ID starting with 08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf not found: ID does not exist" containerID="08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf"
Nov 02 14:10:36.852487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.852445    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf"} err="failed to get container status \"08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf\": rpc error: code = NotFound desc = could not find container \"08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf\": container with ID starting with 08e9ab5794c73c58915ed390899dd0c83f05b91c3c1acec47c6e3f92c63470bf not found: ID does not exist"
Nov 02 14:10:36.893827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.893801    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"]
Nov 02 14:10:36.907117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:36.907094    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-t29v8"]
Nov 02 14:10:37.403810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:37.403743    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" path="/var/lib/kubelet/pods/7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3/volumes"
Nov 02 14:10:38.366870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:38.366837    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"]
Nov 02 14:10:38.367374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:38.367090    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy" containerID="cri-o://b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00" gracePeriod=30
Nov 02 14:10:39.024802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:39.024750    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:10:39.024980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:39.024815    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:10:43.627752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.627723    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:10:43.775401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775359    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-credential-socket\") pod \"5048905b-cee0-4c96-ab97-a91a67d749db\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") "
Nov 02 14:10:43.775599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775415    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-socket\") pod \"5048905b-cee0-4c96-ab97-a91a67d749db\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") "
Nov 02 14:10:43.775670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775606    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-envoy\") pod \"5048905b-cee0-4c96-ab97-a91a67d749db\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") "
Nov 02 14:10:43.775670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775650    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-istio-token\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-istio-token\") pod \"5048905b-cee0-4c96-ab97-a91a67d749db\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") "
Nov 02 14:10:43.775826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775643    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-credential-socket" pod "5048905b-cee0-4c96-ab97-a91a67d749db" (UID: "5048905b-cee0-4c96-ab97-a91a67d749db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:43.775826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775692    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-certs\") pod \"5048905b-cee0-4c96-ab97-a91a67d749db\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") "
Nov 02 14:10:43.775826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775698    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-socket" pod "5048905b-cee0-4c96-ab97-a91a67d749db" (UID: "5048905b-cee0-4c96-ab97-a91a67d749db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:43.775826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775739    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-data\") pod \"5048905b-cee0-4c96-ab97-a91a67d749db\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") "
Nov 02 14:10:43.775826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775772    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-kube-api-access-cqszt\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-kube-api-access-cqszt\") pod \"5048905b-cee0-4c96-ab97-a91a67d749db\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") "
Nov 02 14:10:43.776109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775857    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5048905b-cee0-4c96-ab97-a91a67d749db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5048905b-cee0-4c96-ab97-a91a67d749db-istiod-ca-cert\") pod \"5048905b-cee0-4c96-ab97-a91a67d749db\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") "
Nov 02 14:10:43.776109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.775941    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5048905b-cee0-4c96-ab97-a91a67d749db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5048905b-cee0-4c96-ab97-a91a67d749db-istio-podinfo\") pod \"5048905b-cee0-4c96-ab97-a91a67d749db\" (UID: \"5048905b-cee0-4c96-ab97-a91a67d749db\") "
Nov 02 14:10:43.776933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.776443    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-data" pod "5048905b-cee0-4c96-ab97-a91a67d749db" (UID: "5048905b-cee0-4c96-ab97-a91a67d749db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:43.777116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.776695    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-certs" pod "5048905b-cee0-4c96-ab97-a91a67d749db" (UID: "5048905b-cee0-4c96-ab97-a91a67d749db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:43.777116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.777056    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5048905b-cee0-4c96-ab97-a91a67d749db-istiod-ca-cert" pod "5048905b-cee0-4c96-ab97-a91a67d749db" (UID: "5048905b-cee0-4c96-ab97-a91a67d749db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:43.777349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.776573    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:43.777349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.777158    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:43.778365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.778336    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-envoy" pod "5048905b-cee0-4c96-ab97-a91a67d749db" (UID: "5048905b-cee0-4c96-ab97-a91a67d749db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:10:43.778502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.778459    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-istio-token" pod "5048905b-cee0-4c96-ab97-a91a67d749db" (UID: "5048905b-cee0-4c96-ab97-a91a67d749db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:43.778996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.778969    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-kube-api-access-cqszt" pod "5048905b-cee0-4c96-ab97-a91a67d749db" (UID: "5048905b-cee0-4c96-ab97-a91a67d749db"). InnerVolumeSpecName "kube-api-access-cqszt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:43.779469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.779449    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5048905b-cee0-4c96-ab97-a91a67d749db-istio-podinfo" pod "5048905b-cee0-4c96-ab97-a91a67d749db" (UID: "5048905b-cee0-4c96-ab97-a91a67d749db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:10:43.878122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.878082    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5048905b-cee0-4c96-ab97-a91a67d749db-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:43.878122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.878115    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:43.878122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.878128    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:43.878366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.878139    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:43.878366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.878151    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5048905b-cee0-4c96-ab97-a91a67d749db-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:43.878366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.878162    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqszt\" (UniqueName: \"kubernetes.io/projected/5048905b-cee0-4c96-ab97-a91a67d749db-kube-api-access-cqszt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:43.878366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.878173    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5048905b-cee0-4c96-ab97-a91a67d749db-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:43.912109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.912072    5034 generic.go:356] "Generic (PLEG): container finished" podID="5048905b-cee0-4c96-ab97-a91a67d749db" containerID="b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00" exitCode=0
Nov 02 14:10:43.912293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.912131    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"
Nov 02 14:10:43.912353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.912294    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" event={"ID":"5048905b-cee0-4c96-ab97-a91a67d749db","Type":"ContainerDied","Data":"b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00"}
Nov 02 14:10:43.912353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.912336    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d" event={"ID":"5048905b-cee0-4c96-ab97-a91a67d749db","Type":"ContainerDied","Data":"398428c4626c4d688f0709ee3f453a916753e1710d4a75459e5a1c62a05e6232"}
Nov 02 14:10:43.912445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.912363    5034 scope.go:122] "RemoveContainer" containerID="b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00"
Nov 02 14:10:43.925192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.925168    5034 scope.go:122] "RemoveContainer" containerID="990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45"
Nov 02 14:10:43.934729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.934707    5034 scope.go:122] "RemoveContainer" containerID="b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00"
Nov 02 14:10:43.935073 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:10:43.935049    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00\": container with ID starting with b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00 not found: ID does not exist" containerID="b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00"
Nov 02 14:10:43.935160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.935085    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00"} err="failed to get container status \"b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00\": rpc error: code = NotFound desc = could not find container \"b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00\": container with ID starting with b4f7adbe47c90a398c46384d387b6f7e04107a38b5308982c445fed1e1ceac00 not found: ID does not exist"
Nov 02 14:10:43.935160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.935123    5034 scope.go:122] "RemoveContainer" containerID="990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45"
Nov 02 14:10:43.935417 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:10:43.935393    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45\": container with ID starting with 990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45 not found: ID does not exist" containerID="990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45"
Nov 02 14:10:43.935480 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.935426    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45"} err="failed to get container status \"990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45\": rpc error: code = NotFound desc = could not find container \"990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45\": container with ID starting with 990a7c8e970d527727b0181ea98896847b7a875070dd91f0b5934bb005755d45 not found: ID does not exist"
Nov 02 14:10:43.970876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.970849    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"]
Nov 02 14:10:43.976608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:43.976580    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b7d400-openshift-default-68bfcf4769-pmx6d"]
Nov 02 14:10:45.400230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:45.400183    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" path="/var/lib/kubelet/pods/5048905b-cee0-4c96-ab97-a91a67d749db/volumes"
Nov 02 14:10:46.938547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:46.938447    5034 generic.go:356] "Generic (PLEG): container finished" podID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerID="d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328" exitCode=0
Nov 02 14:10:46.938547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:46.938517    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" event={"ID":"8f666d7f-79d0-4556-b859-62fd24beb2a0","Type":"ContainerDied","Data":"d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328"}
Nov 02 14:10:46.939040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:46.938894    5034 scope.go:122] "RemoveContainer" containerID="d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328"
Nov 02 14:10:46.940370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:46.940341    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:10:46.940472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:46.940404    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:10:46.940472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:46.940437    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:10:47.947427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:47.947395    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" event={"ID":"8f666d7f-79d0-4556-b859-62fd24beb2a0","Type":"ContainerStarted","Data":"fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5"}
Nov 02 14:10:48.359924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:48.359883    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:10:48.360102 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:48.360047    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:10:48.360162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:48.360135    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 14:10:48.429391 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:10:48.429352    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:10:49.371252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:49.371215    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:10:49.371852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:49.371821    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 14:10:50.359912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:50.359878    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:10:50.360121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:50.359948    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 14:10:51.359896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:51.359858    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:10:51.360295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:51.359922    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 14:10:51.807381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:51.807332    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"]
Nov 02 14:10:51.823547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:51.823508    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"]
Nov 02 14:10:52.360682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:52.360652    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Nov 02 14:10:52.361248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:52.360701    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Nov 02 14:10:53.441060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:53.441025    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:10:53.441518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:53.441498    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:10:53.442281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:53.442260    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:10:56.280831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:56.280078    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt_3dc1483e-0366-4104-90ea-3c1cbaa0de70/payload-processing/0.log"
Nov 02 14:10:56.808362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:56.808286    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t" podUID="bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7" containerName="payload-processing" containerID="cri-o://de42bd82d3b29289ec0dbcb8b1a6d1e73864959ce1be579ead460ff58782773c" gracePeriod=25
Nov 02 14:10:56.824113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:56.824079    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb" podUID="f9456c6a-5a91-43ac-b70e-74b1b540f800" containerName="payload-pre-processing" containerID="cri-o://d50a4c9de70682d16f92d4cd457035082ead03d0c550a61dde6702b91df8f464" gracePeriod=25
Nov 02 14:10:57.057016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.056984    5034 generic.go:356] "Generic (PLEG): container finished" podID="bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7" containerID="de42bd82d3b29289ec0dbcb8b1a6d1e73864959ce1be579ead460ff58782773c" exitCode=0
Nov 02 14:10:57.057195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.057062    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t" event={"ID":"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7","Type":"ContainerDied","Data":"de42bd82d3b29289ec0dbcb8b1a6d1e73864959ce1be579ead460ff58782773c"}
Nov 02 14:10:57.059698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.059633    5034 generic.go:356] "Generic (PLEG): container finished" podID="f9456c6a-5a91-43ac-b70e-74b1b540f800" containerID="d50a4c9de70682d16f92d4cd457035082ead03d0c550a61dde6702b91df8f464" exitCode=0
Nov 02 14:10:57.059698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.059691    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb" event={"ID":"f9456c6a-5a91-43ac-b70e-74b1b540f800","Type":"ContainerDied","Data":"d50a4c9de70682d16f92d4cd457035082ead03d0c550a61dde6702b91df8f464"}
Nov 02 14:10:57.215399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.215369    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:10:57.256712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.256681    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-kube-api-access-wgljc\" (UniqueName: \"kubernetes.io/projected/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-kube-api-access-wgljc\") pod \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\" (UID: \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\") "
Nov 02 14:10:57.257070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.257045    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-plugins-config-volume\") pod \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\" (UID: \"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7\") "
Nov 02 14:10:57.257404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.257382    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-plugins-config-volume" pod "bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7" (UID: "bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:57.258442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.258414    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:57.263574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.260861    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-kube-api-access-wgljc" pod "bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7" (UID: "bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7"). InnerVolumeSpecName "kube-api-access-wgljc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:57.264132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.264111    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:10:57.358950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.358874    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9456c6a-5a91-43ac-b70e-74b1b540f800-kube-api-access-vnzss\" (UniqueName: \"kubernetes.io/projected/f9456c6a-5a91-43ac-b70e-74b1b540f800-kube-api-access-vnzss\") pod \"f9456c6a-5a91-43ac-b70e-74b1b540f800\" (UID: \"f9456c6a-5a91-43ac-b70e-74b1b540f800\") "
Nov 02 14:10:57.359412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.358997    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9456c6a-5a91-43ac-b70e-74b1b540f800-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9456c6a-5a91-43ac-b70e-74b1b540f800-plugins-config-volume\") pod \"f9456c6a-5a91-43ac-b70e-74b1b540f800\" (UID: \"f9456c6a-5a91-43ac-b70e-74b1b540f800\") "
Nov 02 14:10:57.359412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.359349    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9456c6a-5a91-43ac-b70e-74b1b540f800-plugins-config-volume" pod "f9456c6a-5a91-43ac-b70e-74b1b540f800" (UID: "f9456c6a-5a91-43ac-b70e-74b1b540f800"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:10:57.359504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.359494    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgljc\" (UniqueName: \"kubernetes.io/projected/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7-kube-api-access-wgljc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:57.359554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.359509    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9456c6a-5a91-43ac-b70e-74b1b540f800-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:57.361130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.361105    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9456c6a-5a91-43ac-b70e-74b1b540f800-kube-api-access-vnzss" pod "f9456c6a-5a91-43ac-b70e-74b1b540f800" (UID: "f9456c6a-5a91-43ac-b70e-74b1b540f800"). InnerVolumeSpecName "kube-api-access-vnzss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:10:57.460609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:57.460584    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnzss\" (UniqueName: \"kubernetes.io/projected/f9456c6a-5a91-43ac-b70e-74b1b540f800-kube-api-access-vnzss\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:10:58.092858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.068855    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"
Nov 02 14:10:58.120418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.068960    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb" event={"ID":"f9456c6a-5a91-43ac-b70e-74b1b540f800","Type":"ContainerDied","Data":"c56618bcb0653a530392205f292909deb59e099f62d2c2d98a520644a4f8e5d0"}
Nov 02 14:10:58.120418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.110146    5034 scope.go:122] "RemoveContainer" containerID="d50a4c9de70682d16f92d4cd457035082ead03d0c550a61dde6702b91df8f464"
Nov 02 14:10:58.120418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.113291    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t" event={"ID":"bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7","Type":"ContainerDied","Data":"06de72dd094eb11d018b5840468843fdf885f979d40c80629e5bef579cfe77cc"}
Nov 02 14:10:58.120418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.113425    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"
Nov 02 14:10:58.136336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.136313    5034 scope.go:122] "RemoveContainer" containerID="de42bd82d3b29289ec0dbcb8b1a6d1e73864959ce1be579ead460ff58782773c"
Nov 02 14:10:58.207835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.206258    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"]
Nov 02 14:10:58.212151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.212123    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1d69c-d4b7cd98f-2zl2t"]
Nov 02 14:10:58.262721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.262686    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"]
Nov 02 14:10:58.303802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:58.296083    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1d69c-9fd77ccb7-jz7rb"]
Nov 02 14:10:59.505799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:59.504511    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7" path="/var/lib/kubelet/pods/bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7/volumes"
Nov 02 14:10:59.505799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:10:59.505253    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9456c6a-5a91-43ac-b70e-74b1b540f800" path="/var/lib/kubelet/pods/f9456c6a-5a91-43ac-b70e-74b1b540f800/volumes"
Nov 02 14:11:06.190685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:06.190588    5034 generic.go:356] "Generic (PLEG): container finished" podID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerID="89990b0326ad6f86c2b4553aa75ae441ff81d5ef643f1e87b69ba8b9ef3c48d4" exitCode=0
Nov 02 14:11:06.191224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:06.190664    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" event={"ID":"a7c3817b-5573-4131-8ec9-59a101e298f8","Type":"ContainerDied","Data":"89990b0326ad6f86c2b4553aa75ae441ff81d5ef643f1e87b69ba8b9ef3c48d4"}
Nov 02 14:11:06.191300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:06.191223    5034 scope.go:122] "RemoveContainer" containerID="89990b0326ad6f86c2b4553aa75ae441ff81d5ef643f1e87b69ba8b9ef3c48d4"
Nov 02 14:11:06.194196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:06.193010    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:11:06.194196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:06.193074    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:11:06.194196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:06.193105    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:11:07.211074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:07.211041    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" event={"ID":"a7c3817b-5573-4131-8ec9-59a101e298f8","Type":"ContainerStarted","Data":"07eccaca6e564c825b701e2b9ed9463bcf94d0f68139b7d3332defe280c23ed1"}
Nov 02 14:11:08.023084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:08.023055    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:11:08.023351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:08.023328    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 14:11:08.023404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:08.023377    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 14:11:09.022375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:09.022342    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 14:11:09.023075 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:09.023042    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 14:11:10.023651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:10.023610    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 14:11:10.024243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:10.023668    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 14:11:11.023230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:11.023192    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 14:11:11.023526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:11.023486    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 14:11:12.022325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:12.022285    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Nov 02 14:11:12.022719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:12.022336    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Nov 02 14:11:13.027338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:13.027303    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:11:13.027828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:13.027808    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:11:13.028487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:13.028467    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:11:14.979023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.978992    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"]
Nov 02 14:11:14.979907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.979886    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7" containerName="payload-processing"
Nov 02 14:11:14.979907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.979908    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7" containerName="payload-processing"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.979925    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.979934    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.979955    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.979963    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.979979    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.979986    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.979997    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ac4633fd-49f3-4cf4-8457-c89cc30aa4dd" containerName="payload-pre-processing"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980004    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4633fd-49f3-4cf4-8457-c89cc30aa4dd" containerName="payload-pre-processing"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980020    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d13007cc-d6c8-447a-9653-d40f74514064" containerName="payload-processing"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980027    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13007cc-d6c8-447a-9653-d40f74514064" containerName="payload-processing"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980037    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980044    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980068    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy"
Nov 02 14:11:14.980096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980076    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980109    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f9456c6a-5a91-43ac-b70e-74b1b540f800" containerName="payload-pre-processing"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980117    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9456c6a-5a91-43ac-b70e-74b1b540f800" containerName="payload-pre-processing"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980284    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980297    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4633fd-49f3-4cf4-8457-c89cc30aa4dd" containerName="payload-pre-processing"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980311    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13007cc-d6c8-447a-9653-d40f74514064" containerName="payload-processing"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980323    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980332    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5048905b-cee0-4c96-ab97-a91a67d749db" containerName="istio-proxy"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980345    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9456c6a-5a91-43ac-b70e-74b1b540f800" containerName="payload-pre-processing"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980361    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf0da69-038d-4fc3-9e8a-2a8501ff7fe7" containerName="payload-processing"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980368    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9df9c2-2c16-4bb5-bb05-ee1667367dba" containerName="istio-proxy"
Nov 02 14:11:14.980810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.980380    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb2bcbb-ceb8-4a51-8ee0-9e8839e832a3" containerName="istio-proxy"
Nov 02 14:11:14.983340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:14.983319    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.004306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.004282    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:11:15.017463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.017443    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:11:15.017759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.017743    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:11:15.017992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.017979    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:11:15.021432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.021405    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"]
Nov 02 14:11:15.091766 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.091732    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ddr\" (UniqueName: \"kubernetes.io/projected/978b6664-2b85-4ec1-99d8-039e3d12fc69-kube-api-access-k9ddr\") pod \"maas-api-revoke-sub-b0a75e93-gld2z\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.091981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.091925    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a75e93-gld2z\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.092054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.092035    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a75e93-gld2z\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.196280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.196244    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ddr\" (UniqueName: \"kubernetes.io/projected/978b6664-2b85-4ec1-99d8-039e3d12fc69-kube-api-access-k9ddr\") pod \"maas-api-revoke-sub-b0a75e93-gld2z\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.197304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.197276    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a75e93-gld2z\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.200094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.200073    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a75e93-gld2z\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.201925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.199937    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a75e93-gld2z\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.204126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.204095    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a75e93-gld2z\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.221551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.221520    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ddr\" (UniqueName: \"kubernetes.io/projected/978b6664-2b85-4ec1-99d8-039e3d12fc69-kube-api-access-k9ddr\") pod \"maas-api-revoke-sub-b0a75e93-gld2z\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.298187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.298151    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:15.603659 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:11:15.603626    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978b6664_2b85_4ec1_99d8_039e3d12fc69.slice/crio-ccde49b997d2fe4ba08a171c839e3d3c60a0c993bc9ef6fbc21cf160a053cd33 WatchSource:0}: Error finding container ccde49b997d2fe4ba08a171c839e3d3c60a0c993bc9ef6fbc21cf160a053cd33: Status 404 returned error can't find the container with id ccde49b997d2fe4ba08a171c839e3d3c60a0c993bc9ef6fbc21cf160a053cd33
Nov 02 14:11:15.604342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:15.604318    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"]
Nov 02 14:11:16.289426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:16.289384    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z" event={"ID":"978b6664-2b85-4ec1-99d8-039e3d12fc69","Type":"ContainerStarted","Data":"90ce572d55bffb4a6b0bfe885921f862e141332cb3413a89562b0706d520d6f5"}
Nov 02 14:11:16.289940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:16.289483    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z" event={"ID":"978b6664-2b85-4ec1-99d8-039e3d12fc69","Type":"ContainerStarted","Data":"ccde49b997d2fe4ba08a171c839e3d3c60a0c993bc9ef6fbc21cf160a053cd33"}
Nov 02 14:11:16.335368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:16.335315    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z" podStartSLOduration=2.335300383 podStartE2EDuration="2.335300383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:16.332102283 +0000 UTC m=+1667.083406720" watchObservedRunningTime="2026-11-02 14:11:16.335300383 +0000 UTC m=+1667.086604839"
Nov 02 14:11:17.301235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:17.301200    5034 generic.go:356] "Generic (PLEG): container finished" podID="978b6664-2b85-4ec1-99d8-039e3d12fc69" containerID="90ce572d55bffb4a6b0bfe885921f862e141332cb3413a89562b0706d520d6f5" exitCode=0
Nov 02 14:11:17.301687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:17.301252    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z" event={"ID":"978b6664-2b85-4ec1-99d8-039e3d12fc69","Type":"ContainerDied","Data":"90ce572d55bffb4a6b0bfe885921f862e141332cb3413a89562b0706d520d6f5"}
Nov 02 14:11:18.506510 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:11:18.506434    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:11:18.565127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.565073    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:18.660328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.660292    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978b6664-2b85-4ec1-99d8-039e3d12fc69-kube-api-access-k9ddr\" (UniqueName: \"kubernetes.io/projected/978b6664-2b85-4ec1-99d8-039e3d12fc69-kube-api-access-k9ddr\") pod \"978b6664-2b85-4ec1-99d8-039e3d12fc69\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") "
Nov 02 14:11:18.660512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.660376    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-xks-ca\") pod \"978b6664-2b85-4ec1-99d8-039e3d12fc69\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") "
Nov 02 14:11:18.660627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.660607    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-openshift-ca\") pod \"978b6664-2b85-4ec1-99d8-039e3d12fc69\" (UID: \"978b6664-2b85-4ec1-99d8-039e3d12fc69\") "
Nov 02 14:11:18.661158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.661124    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-openshift-ca" pod "978b6664-2b85-4ec1-99d8-039e3d12fc69" (UID: "978b6664-2b85-4ec1-99d8-039e3d12fc69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:18.663169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.663141    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978b6664-2b85-4ec1-99d8-039e3d12fc69-kube-api-access-k9ddr" pod "978b6664-2b85-4ec1-99d8-039e3d12fc69" (UID: "978b6664-2b85-4ec1-99d8-039e3d12fc69"). InnerVolumeSpecName "kube-api-access-k9ddr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:18.663268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.663206    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-xks-ca" pod "978b6664-2b85-4ec1-99d8-039e3d12fc69" (UID: "978b6664-2b85-4ec1-99d8-039e3d12fc69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:18.663905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.663888    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:18.765156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.765125    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9ddr\" (UniqueName: \"kubernetes.io/projected/978b6664-2b85-4ec1-99d8-039e3d12fc69-kube-api-access-k9ddr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:18.765156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:18.765153    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/978b6664-2b85-4ec1-99d8-039e3d12fc69-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:19.318834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:19.318804    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z" event={"ID":"978b6664-2b85-4ec1-99d8-039e3d12fc69","Type":"ContainerDied","Data":"ccde49b997d2fe4ba08a171c839e3d3c60a0c993bc9ef6fbc21cf160a053cd33"}
Nov 02 14:11:19.318834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:19.318826    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"
Nov 02 14:11:19.319048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:19.318851    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccde49b997d2fe4ba08a171c839e3d3c60a0c993bc9ef6fbc21cf160a053cd33"
Nov 02 14:11:30.136802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:30.136758    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"]
Nov 02 14:11:30.137825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:30.137053    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy" containerID="cri-o://de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08" gracePeriod=30
Nov 02 14:11:33.395486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:33.395459    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"]
Nov 02 14:11:34.576911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:34.576871    5034 patch_prober.go:29] interesting pod/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:11:34.577394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:34.576932    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:11:35.213545 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:11:35.213518    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f724a8_1fa4_48c1_aace_016fd015dbac.slice/crio-de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 14:11:35.428916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.428887    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 14:11:35.434244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.434217    5034 generic.go:356] "Generic (PLEG): container finished" podID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerID="de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08" exitCode=0
Nov 02 14:11:35.434383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.434259    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"
Nov 02 14:11:35.434383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.434297    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" event={"ID":"90f724a8-1fa4-48c1-aace-016fd015dbac","Type":"ContainerDied","Data":"de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08"}
Nov 02 14:11:35.434383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.434352    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq" event={"ID":"90f724a8-1fa4-48c1-aace-016fd015dbac","Type":"ContainerDied","Data":"347c8b3641c7c7510e700c27916d6b11ec16b01cf71dd400b79b9d3e0c7b5d8b"}
Nov 02 14:11:35.434383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.434381    5034 scope.go:122] "RemoveContainer" containerID="de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08"
Nov 02 14:11:35.448531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.448511    5034 scope.go:122] "RemoveContainer" containerID="de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08"
Nov 02 14:11:35.448829 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:11:35.448797    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08\": container with ID starting with de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08 not found: ID does not exist" containerID="de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08"
Nov 02 14:11:35.448953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.448835    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08"} err="failed to get container status \"de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08\": rpc error: code = NotFound desc = could not find container \"de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08\": container with ID starting with de7c31b33cc534c67e05a4224a7178b0ea13c02b852487d3b849909fd1716e08 not found: ID does not exist"
Nov 02 14:11:35.547539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547505    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-certs\") pod \"90f724a8-1fa4-48c1-aace-016fd015dbac\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") "
Nov 02 14:11:35.547730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547567    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-envoy\") pod \"90f724a8-1fa4-48c1-aace-016fd015dbac\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") "
Nov 02 14:11:35.547730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547589    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f724a8-1fa4-48c1-aace-016fd015dbac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f724a8-1fa4-48c1-aace-016fd015dbac-istiod-ca-cert\") pod \"90f724a8-1fa4-48c1-aace-016fd015dbac\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") "
Nov 02 14:11:35.547730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547626    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-podinfo\") pod \"90f724a8-1fa4-48c1-aace-016fd015dbac\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") "
Nov 02 14:11:35.547730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547676    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-data\") pod \"90f724a8-1fa4-48c1-aace-016fd015dbac\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") "
Nov 02 14:11:35.547730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547706    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-token\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-token\") pod \"90f724a8-1fa4-48c1-aace-016fd015dbac\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") "
Nov 02 14:11:35.548078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547746    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-kube-api-access-lvjvp\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-kube-api-access-lvjvp\") pod \"90f724a8-1fa4-48c1-aace-016fd015dbac\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") "
Nov 02 14:11:35.548078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547764    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-certs" pod "90f724a8-1fa4-48c1-aace-016fd015dbac" (UID: "90f724a8-1fa4-48c1-aace-016fd015dbac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:35.548078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547849    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-credential-socket\") pod \"90f724a8-1fa4-48c1-aace-016fd015dbac\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") "
Nov 02 14:11:35.548078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.547969    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-socket\") pod \"90f724a8-1fa4-48c1-aace-016fd015dbac\" (UID: \"90f724a8-1fa4-48c1-aace-016fd015dbac\") "
Nov 02 14:11:35.548078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.548055    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-data" pod "90f724a8-1fa4-48c1-aace-016fd015dbac" (UID: "90f724a8-1fa4-48c1-aace-016fd015dbac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:35.548078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.548061    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f724a8-1fa4-48c1-aace-016fd015dbac-istiod-ca-cert" pod "90f724a8-1fa4-48c1-aace-016fd015dbac" (UID: "90f724a8-1fa4-48c1-aace-016fd015dbac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:35.548402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.548254    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-credential-socket" pod "90f724a8-1fa4-48c1-aace-016fd015dbac" (UID: "90f724a8-1fa4-48c1-aace-016fd015dbac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:35.548402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.548311    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-socket" pod "90f724a8-1fa4-48c1-aace-016fd015dbac" (UID: "90f724a8-1fa4-48c1-aace-016fd015dbac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:35.548679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.548659    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:35.548679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.548678    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:35.548925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.548687    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90f724a8-1fa4-48c1-aace-016fd015dbac-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:35.548925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.548696    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:35.548925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.548703    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:35.550244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.550216    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-envoy" pod "90f724a8-1fa4-48c1-aace-016fd015dbac" (UID: "90f724a8-1fa4-48c1-aace-016fd015dbac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:35.550244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.550205    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-token" pod "90f724a8-1fa4-48c1-aace-016fd015dbac" (UID: "90f724a8-1fa4-48c1-aace-016fd015dbac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:35.550447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.550430    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-podinfo" pod "90f724a8-1fa4-48c1-aace-016fd015dbac" (UID: "90f724a8-1fa4-48c1-aace-016fd015dbac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:11:35.550540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.550518    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-kube-api-access-lvjvp" pod "90f724a8-1fa4-48c1-aace-016fd015dbac" (UID: "90f724a8-1fa4-48c1-aace-016fd015dbac"). InnerVolumeSpecName "kube-api-access-lvjvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:35.662031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.661998    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:35.662523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.662033    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvjvp\" (UniqueName: \"kubernetes.io/projected/90f724a8-1fa4-48c1-aace-016fd015dbac-kube-api-access-lvjvp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:35.662523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.662051    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:35.662523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.662064    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90f724a8-1fa4-48c1-aace-016fd015dbac-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:35.800883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.800844    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"]
Nov 02 14:11:35.813821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:35.813791    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-d5ndq"]
Nov 02 14:11:37.397994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:37.397952    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" path="/var/lib/kubelet/pods/90f724a8-1fa4-48c1-aace-016fd015dbac/volumes"
Nov 02 14:11:37.852154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:37.852117    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"]
Nov 02 14:11:37.852370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:37.852328    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x" podUID="d80a5c73-5ac2-45cb-9a76-ce646d16fb2b" containerName="payload-processing" containerID="cri-o://5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421" gracePeriod=30
Nov 02 14:11:38.150640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.150615    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:11:38.252802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.245071    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-plugins-config-volume\") pod \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\" (UID: \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\") "
Nov 02 14:11:38.253309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.253275    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-kube-api-access-wbbwp\" (UniqueName: \"kubernetes.io/projected/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-kube-api-access-wbbwp\") pod \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\" (UID: \"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b\") "
Nov 02 14:11:38.265125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.265087    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-kube-api-access-wbbwp" pod "d80a5c73-5ac2-45cb-9a76-ce646d16fb2b" (UID: "d80a5c73-5ac2-45cb-9a76-ce646d16fb2b"). InnerVolumeSpecName "kube-api-access-wbbwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:38.273625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.273587    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-plugins-config-volume" pod "d80a5c73-5ac2-45cb-9a76-ce646d16fb2b" (UID: "d80a5c73-5ac2-45cb-9a76-ce646d16fb2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:38.356675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.356635    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:38.356675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.356675    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbbwp\" (UniqueName: \"kubernetes.io/projected/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b-kube-api-access-wbbwp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:38.396067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.395981    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r" podUID="f4f30da4-caef-4251-bcd1-9c242bef4d4b" containerName="payload-pre-processing" containerID="cri-o://ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a" gracePeriod=25
Nov 02 14:11:38.455204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.455176    5034 generic.go:356] "Generic (PLEG): container finished" podID="d80a5c73-5ac2-45cb-9a76-ce646d16fb2b" containerID="5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421" exitCode=0
Nov 02 14:11:38.455607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.455219    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"
Nov 02 14:11:38.455607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.455269    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x" event={"ID":"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b","Type":"ContainerDied","Data":"5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421"}
Nov 02 14:11:38.455607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.455339    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x" event={"ID":"d80a5c73-5ac2-45cb-9a76-ce646d16fb2b","Type":"ContainerDied","Data":"0b08e3a11da87d7bfb0bb71321b1fbdd830695aacaaf2817322dca2cbc3208a1"}
Nov 02 14:11:38.455607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.455372    5034 scope.go:122] "RemoveContainer" containerID="5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421"
Nov 02 14:11:38.510606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.510585    5034 scope.go:122] "RemoveContainer" containerID="5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421"
Nov 02 14:11:38.510967 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:11:38.510938    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421\": container with ID starting with 5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421 not found: ID does not exist" containerID="5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421"
Nov 02 14:11:38.511054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.510977    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421"} err="failed to get container status \"5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421\": rpc error: code = NotFound desc = could not find container \"5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421\": container with ID starting with 5bea2b45596161b5ec83f34466db825b4e32bcd6c3b3347c0f4e197ecdd11421 not found: ID does not exist"
Nov 02 14:11:38.522084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.522058    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"]
Nov 02 14:11:38.525215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.525192    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-13367a-6dbb64c5c6-ggf9x"]
Nov 02 14:11:38.642666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.642631    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"]
Nov 02 14:11:38.643342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.643319    5034 cpu_manager.go:414] "containerMap: removing container" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy"
Nov 02 14:11:38.643342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.643340    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy"
Nov 02 14:11:38.643505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.643366    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d80a5c73-5ac2-45cb-9a76-ce646d16fb2b" containerName="payload-processing"
Nov 02 14:11:38.643505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.643372    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80a5c73-5ac2-45cb-9a76-ce646d16fb2b" containerName="payload-processing"
Nov 02 14:11:38.643505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.643391    5034 cpu_manager.go:414] "containerMap: removing container" podUID="978b6664-2b85-4ec1-99d8-039e3d12fc69" containerName="revoke-keys"
Nov 02 14:11:38.643505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.643398    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978b6664-2b85-4ec1-99d8-039e3d12fc69" containerName="revoke-keys"
Nov 02 14:11:38.643739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.643510    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80a5c73-5ac2-45cb-9a76-ce646d16fb2b" containerName="payload-processing"
Nov 02 14:11:38.643739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.643522    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="978b6664-2b85-4ec1-99d8-039e3d12fc69" containerName="revoke-keys"
Nov 02 14:11:38.643739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.643532    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f724a8-1fa4-48c1-aace-016fd015dbac" containerName="istio-proxy"
Nov 02 14:11:38.646486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.646431    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.660410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.660389    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4dbe85e-p9gsr\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.660495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.660464    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5k7\" (UniqueName: \"kubernetes.io/projected/5e23929a-567b-4366-951d-d0fc22ca5b08-kube-api-access-vg5k7\") pod \"maas-api-revoke-sub-d4dbe85e-p9gsr\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.660542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.660521    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4dbe85e-p9gsr\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.664304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.664286    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:11:38.667323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.667304    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:11:38.667416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.667400    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:11:38.675063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.673224    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"]
Nov 02 14:11:38.737861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.737827    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:11:38.738033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.737967    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:11:38.762032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.762002    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f30da4-caef-4251-bcd1-9c242bef4d4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f30da4-caef-4251-bcd1-9c242bef4d4b-plugins-config-volume\") pod \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\" (UID: \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\") "
Nov 02 14:11:38.762226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.762096    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f30da4-caef-4251-bcd1-9c242bef4d4b-kube-api-access-9zvns\" (UniqueName: \"kubernetes.io/projected/f4f30da4-caef-4251-bcd1-9c242bef4d4b-kube-api-access-9zvns\") pod \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\" (UID: \"f4f30da4-caef-4251-bcd1-9c242bef4d4b\") "
Nov 02 14:11:38.762457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.762434    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f30da4-caef-4251-bcd1-9c242bef4d4b-plugins-config-volume" pod "f4f30da4-caef-4251-bcd1-9c242bef4d4b" (UID: "f4f30da4-caef-4251-bcd1-9c242bef4d4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:38.762561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.762543    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4dbe85e-p9gsr\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.762689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.762669    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5k7\" (UniqueName: \"kubernetes.io/projected/5e23929a-567b-4366-951d-d0fc22ca5b08-kube-api-access-vg5k7\") pod \"maas-api-revoke-sub-d4dbe85e-p9gsr\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.763413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.763390    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4dbe85e-p9gsr\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.763859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.763484    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f30da4-caef-4251-bcd1-9c242bef4d4b-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:38.764121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.764094    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4dbe85e-p9gsr\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.764600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.764570    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f30da4-caef-4251-bcd1-9c242bef4d4b-kube-api-access-9zvns" pod "f4f30da4-caef-4251-bcd1-9c242bef4d4b" (UID: "f4f30da4-caef-4251-bcd1-9c242bef4d4b"). InnerVolumeSpecName "kube-api-access-9zvns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:38.766094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.766061    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4dbe85e-p9gsr\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.788651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.788565    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5k7\" (UniqueName: \"kubernetes.io/projected/5e23929a-567b-4366-951d-d0fc22ca5b08-kube-api-access-vg5k7\") pod \"maas-api-revoke-sub-d4dbe85e-p9gsr\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:38.865154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.865122    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvns\" (UniqueName: \"kubernetes.io/projected/f4f30da4-caef-4251-bcd1-9c242bef4d4b-kube-api-access-9zvns\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:38.963443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:38.963368    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:39.279994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.277404    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"]
Nov 02 14:11:39.397067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.397038    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80a5c73-5ac2-45cb-9a76-ce646d16fb2b" path="/var/lib/kubelet/pods/d80a5c73-5ac2-45cb-9a76-ce646d16fb2b/volumes"
Nov 02 14:11:39.463519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.463490    5034 generic.go:356] "Generic (PLEG): container finished" podID="f4f30da4-caef-4251-bcd1-9c242bef4d4b" containerID="ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a" exitCode=0
Nov 02 14:11:39.463954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.463537    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"
Nov 02 14:11:39.463954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.463571    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r" event={"ID":"f4f30da4-caef-4251-bcd1-9c242bef4d4b","Type":"ContainerDied","Data":"ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a"}
Nov 02 14:11:39.463954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.463617    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r" event={"ID":"f4f30da4-caef-4251-bcd1-9c242bef4d4b","Type":"ContainerDied","Data":"069f74a1167092986c64e122e32585078fd3bd3898a359047bdad8e15c3dd98c"}
Nov 02 14:11:39.463954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.463639    5034 scope.go:122] "RemoveContainer" containerID="ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a"
Nov 02 14:11:39.465975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.465511    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr" event={"ID":"5e23929a-567b-4366-951d-d0fc22ca5b08","Type":"ContainerStarted","Data":"2b15be6c01dec4ac8ba04d41bd6d1ee5f3430c6dfe2aeff3fd1081e586818151"}
Nov 02 14:11:39.480194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.480169    5034 scope.go:122] "RemoveContainer" containerID="ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a"
Nov 02 14:11:39.480523 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:11:39.480497    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a\": container with ID starting with ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a not found: ID does not exist" containerID="ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a"
Nov 02 14:11:39.480622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.480531    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a"} err="failed to get container status \"ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a\": rpc error: code = NotFound desc = could not find container \"ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a\": container with ID starting with ed810ffecfebb7f6bcecfececa34b0640560370bd31f71f129586a5bb7785f8a not found: ID does not exist"
Nov 02 14:11:39.525178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.525119    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"]
Nov 02 14:11:39.532031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:39.532009    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-370905-54bfb76655-bwq6r"]
Nov 02 14:11:40.478974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:40.478895    5034 generic.go:356] "Generic (PLEG): container finished" podID="5e23929a-567b-4366-951d-d0fc22ca5b08" containerID="cb9e9c0333465e222304b5a476d10e90f028743999ddf86740821269e8687d67" exitCode=0
Nov 02 14:11:40.479404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:40.478970    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr" event={"ID":"5e23929a-567b-4366-951d-d0fc22ca5b08","Type":"ContainerDied","Data":"cb9e9c0333465e222304b5a476d10e90f028743999ddf86740821269e8687d67"}
Nov 02 14:11:41.047699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.047664    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"]
Nov 02 14:11:41.048301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.048285    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f4f30da4-caef-4251-bcd1-9c242bef4d4b" containerName="payload-pre-processing"
Nov 02 14:11:41.048362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.048302    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f30da4-caef-4251-bcd1-9c242bef4d4b" containerName="payload-pre-processing"
Nov 02 14:11:41.048425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.048416    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f30da4-caef-4251-bcd1-9c242bef4d4b" containerName="payload-pre-processing"
Nov 02 14:11:41.050863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.050842    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.074856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.074831    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"]
Nov 02 14:11:41.127747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.127704    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a4e90dd-5r4m2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.127925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.127810    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7wg\" (UniqueName: \"kubernetes.io/projected/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-kube-api-access-zh7wg\") pod \"maas-api-revoke-sub-7a4e90dd-5r4m2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.128033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.128019    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a4e90dd-5r4m2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.230132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.230096    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a4e90dd-5r4m2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.230318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.230179    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7wg\" (UniqueName: \"kubernetes.io/projected/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-kube-api-access-zh7wg\") pod \"maas-api-revoke-sub-7a4e90dd-5r4m2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.230401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.230381    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a4e90dd-5r4m2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.231090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.231061    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a4e90dd-5r4m2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.233188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.233164    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a4e90dd-5r4m2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.255144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.255117    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7wg\" (UniqueName: \"kubernetes.io/projected/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-kube-api-access-zh7wg\") pod \"maas-api-revoke-sub-7a4e90dd-5r4m2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.365568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.365492    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:41.405353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.405309    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f30da4-caef-4251-bcd1-9c242bef4d4b" path="/var/lib/kubelet/pods/f4f30da4-caef-4251-bcd1-9c242bef4d4b/volumes"
Nov 02 14:11:41.674057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.674032    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:41.698733 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:11:41.698704    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e4cd2b_d057_43f2_91ce_fe2da8a72ec2.slice/crio-ccdc988b0c543153fd69703fd1eaa87208b4fc87c36d72e1b15817d77b5b7f6a WatchSource:0}: Error finding container ccdc988b0c543153fd69703fd1eaa87208b4fc87c36d72e1b15817d77b5b7f6a: Status 404 returned error can't find the container with id ccdc988b0c543153fd69703fd1eaa87208b4fc87c36d72e1b15817d77b5b7f6a
Nov 02 14:11:41.699726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.699701    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"]
Nov 02 14:11:41.843308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.843266    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-xks-ca\") pod \"5e23929a-567b-4366-951d-d0fc22ca5b08\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") "
Nov 02 14:11:41.843504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.843472    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-openshift-ca\") pod \"5e23929a-567b-4366-951d-d0fc22ca5b08\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") "
Nov 02 14:11:41.843585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.843558    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e23929a-567b-4366-951d-d0fc22ca5b08-kube-api-access-vg5k7\" (UniqueName: \"kubernetes.io/projected/5e23929a-567b-4366-951d-d0fc22ca5b08-kube-api-access-vg5k7\") pod \"5e23929a-567b-4366-951d-d0fc22ca5b08\" (UID: \"5e23929a-567b-4366-951d-d0fc22ca5b08\") "
Nov 02 14:11:41.843933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.843900    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-openshift-ca" pod "5e23929a-567b-4366-951d-d0fc22ca5b08" (UID: "5e23929a-567b-4366-951d-d0fc22ca5b08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:41.844264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.844240    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:41.870531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.870497    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-xks-ca" pod "5e23929a-567b-4366-951d-d0fc22ca5b08" (UID: "5e23929a-567b-4366-951d-d0fc22ca5b08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:41.870531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.870521    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e23929a-567b-4366-951d-d0fc22ca5b08-kube-api-access-vg5k7" pod "5e23929a-567b-4366-951d-d0fc22ca5b08" (UID: "5e23929a-567b-4366-951d-d0fc22ca5b08"). InnerVolumeSpecName "kube-api-access-vg5k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:41.945490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.945414    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5k7\" (UniqueName: \"kubernetes.io/projected/5e23929a-567b-4366-951d-d0fc22ca5b08-kube-api-access-vg5k7\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:41.945490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:41.945443    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e23929a-567b-4366-951d-d0fc22ca5b08-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:42.531906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:42.527768    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr" event={"ID":"5e23929a-567b-4366-951d-d0fc22ca5b08","Type":"ContainerDied","Data":"2b15be6c01dec4ac8ba04d41bd6d1ee5f3430c6dfe2aeff3fd1081e586818151"}
Nov 02 14:11:42.543897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:42.543863    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b15be6c01dec4ac8ba04d41bd6d1ee5f3430c6dfe2aeff3fd1081e586818151"
Nov 02 14:11:42.544209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:42.544189    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"
Nov 02 14:11:42.560042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:42.559866    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2" event={"ID":"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2","Type":"ContainerStarted","Data":"12f58e0e865b4888c0eff7b01d65e6c8f6a939bb881a769f82baadad8b1ba10b"}
Nov 02 14:11:42.560042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:42.559939    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2" event={"ID":"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2","Type":"ContainerStarted","Data":"ccdc988b0c543153fd69703fd1eaa87208b4fc87c36d72e1b15817d77b5b7f6a"}
Nov 02 14:11:42.617907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:42.617847    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2" podStartSLOduration=1.6178282099999999 podStartE2EDuration="1.61782821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:42.611638164 +0000 UTC m=+1693.362942604" watchObservedRunningTime="2026-11-02 14:11:42.61782821 +0000 UTC m=+1693.369132650"
Nov 02 14:11:43.568222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:43.568190    5034 generic.go:356] "Generic (PLEG): container finished" podID="82e4cd2b-d057-43f2-91ce-fe2da8a72ec2" containerID="12f58e0e865b4888c0eff7b01d65e6c8f6a939bb881a769f82baadad8b1ba10b" exitCode=0
Nov 02 14:11:43.568631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:43.568248    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2" event={"ID":"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2","Type":"ContainerDied","Data":"12f58e0e865b4888c0eff7b01d65e6c8f6a939bb881a769f82baadad8b1ba10b"}
Nov 02 14:11:43.945270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:43.945163    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"]
Nov 02 14:11:43.945468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:43.945422    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5" podUID="d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" containerName="discovery" containerID="cri-o://8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91" gracePeriod=30
Nov 02 14:11:44.249508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.249480    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:11:44.351448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.351418    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-cacerts\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-cacerts\") pod \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") "
Nov 02 14:11:44.351617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.351497    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-dns-cert\") pod \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") "
Nov 02 14:11:44.351617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.351547    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-token\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-token\") pod \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") "
Nov 02 14:11:44.351617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.351614    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-kubeconfig\") pod \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") "
Nov 02 14:11:44.351853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.351665    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-local-certs\") pod \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") "
Nov 02 14:11:44.351853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.351690    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-kube-api-access-gvxxn\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-kube-api-access-gvxxn\") pod \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") "
Nov 02 14:11:44.351853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.351720    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-ca-configmap\") pod \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\" (UID: \"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7\") "
Nov 02 14:11:44.352681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.352618    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-ca-configmap" pod "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" (UID: "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:44.355026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.354998    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-dns-cert" pod "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" (UID: "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:44.373642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.373549    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-kubeconfig" pod "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" (UID: "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:44.373642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.373563    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-local-certs" pod "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" (UID: "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:11:44.373642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.373605    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-kube-api-access-gvxxn" pod "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" (UID: "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7"). InnerVolumeSpecName "kube-api-access-gvxxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:44.373642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.373613    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-cacerts" pod "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" (UID: "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:44.375254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.375234    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-token" pod "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" (UID: "d818ae64-0d7e-4a50-8200-b77e0dcdf8e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:44.453695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.453660    5034 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-cacerts\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:44.453695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.453695    5034 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-dns-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:44.453948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.453709    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:44.453948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.453721    5034 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-kubeconfig\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:44.453948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.453732    5034 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-local-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:44.453948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.453743    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvxxn\" (UniqueName: \"kubernetes.io/projected/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-kube-api-access-gvxxn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:44.453948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.453757    5034 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7-istio-csr-ca-configmap\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:44.577276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.577239    5034 generic.go:356] "Generic (PLEG): container finished" podID="d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" containerID="8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91" exitCode=0
Nov 02 14:11:44.577731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.577289    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"
Nov 02 14:11:44.577731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.577328    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5" event={"ID":"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7","Type":"ContainerDied","Data":"8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91"}
Nov 02 14:11:44.577731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.577393    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5" event={"ID":"d818ae64-0d7e-4a50-8200-b77e0dcdf8e7","Type":"ContainerDied","Data":"72e9eecff9744dbd92cd46da80d21ae953292a37a240db91394cd04ce38ce446"}
Nov 02 14:11:44.577731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.577428    5034 scope.go:122] "RemoveContainer" containerID="8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91"
Nov 02 14:11:44.590177 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.590154    5034 scope.go:122] "RemoveContainer" containerID="8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91"
Nov 02 14:11:44.590543 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:11:44.590517    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91\": container with ID starting with 8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91 not found: ID does not exist" containerID="8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91"
Nov 02 14:11:44.590627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.590554    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91"} err="failed to get container status \"8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91\": rpc error: code = NotFound desc = could not find container \"8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91\": container with ID starting with 8b043933f3274fd289a51385c75d4ace9cda5ced73aca6c6242c4f50a7e55e91 not found: ID does not exist"
Nov 02 14:11:44.647407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.644082    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"]
Nov 02 14:11:44.657794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.657755    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qxkc5"]
Nov 02 14:11:44.724464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.724438    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:44.761383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.761357    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-kube-api-access-zh7wg\" (UniqueName: \"kubernetes.io/projected/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-kube-api-access-zh7wg\") pod \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") "
Nov 02 14:11:44.761564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.761544    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-openshift-ca\") pod \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") "
Nov 02 14:11:44.761633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.761600    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-xks-ca\") pod \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\" (UID: \"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2\") "
Nov 02 14:11:44.761908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.761880    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-openshift-ca" pod "82e4cd2b-d057-43f2-91ce-fe2da8a72ec2" (UID: "82e4cd2b-d057-43f2-91ce-fe2da8a72ec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:44.762184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.762164    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:44.763461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.763438    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-kube-api-access-zh7wg" pod "82e4cd2b-d057-43f2-91ce-fe2da8a72ec2" (UID: "82e4cd2b-d057-43f2-91ce-fe2da8a72ec2"). InnerVolumeSpecName "kube-api-access-zh7wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:44.763600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.763578    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-xks-ca" pod "82e4cd2b-d057-43f2-91ce-fe2da8a72ec2" (UID: "82e4cd2b-d057-43f2-91ce-fe2da8a72ec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:44.863303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.863228    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:44.863303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:44.863263    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7wg\" (UniqueName: \"kubernetes.io/projected/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2-kube-api-access-zh7wg\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:45.394992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:45.394959    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" path="/var/lib/kubelet/pods/d818ae64-0d7e-4a50-8200-b77e0dcdf8e7/volumes"
Nov 02 14:11:45.586569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:45.586542    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"
Nov 02 14:11:45.587081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:45.586543    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2" event={"ID":"82e4cd2b-d057-43f2-91ce-fe2da8a72ec2","Type":"ContainerDied","Data":"ccdc988b0c543153fd69703fd1eaa87208b4fc87c36d72e1b15817d77b5b7f6a"}
Nov 02 14:11:45.587081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:45.586659    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccdc988b0c543153fd69703fd1eaa87208b4fc87c36d72e1b15817d77b5b7f6a"
Nov 02 14:11:48.413612 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:11:48.413581    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:11:51.630011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.629974    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"]
Nov 02 14:11:51.630633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.630619    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" containerName="discovery"
Nov 02 14:11:51.630687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.630634    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" containerName="discovery"
Nov 02 14:11:51.630687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.630648    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5e23929a-567b-4366-951d-d0fc22ca5b08" containerName="revoke-keys"
Nov 02 14:11:51.630687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.630653    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e23929a-567b-4366-951d-d0fc22ca5b08" containerName="revoke-keys"
Nov 02 14:11:51.630687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.630665    5034 cpu_manager.go:414] "containerMap: removing container" podUID="82e4cd2b-d057-43f2-91ce-fe2da8a72ec2" containerName="revoke-keys"
Nov 02 14:11:51.630687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.630671    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e4cd2b-d057-43f2-91ce-fe2da8a72ec2" containerName="revoke-keys"
Nov 02 14:11:51.630869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.630756    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e23929a-567b-4366-951d-d0fc22ca5b08" containerName="revoke-keys"
Nov 02 14:11:51.630869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.630767    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e4cd2b-d057-43f2-91ce-fe2da8a72ec2" containerName="revoke-keys"
Nov 02 14:11:51.630869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.630798    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d818ae64-0d7e-4a50-8200-b77e0dcdf8e7" containerName="discovery"
Nov 02 14:11:51.634209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.634193    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.653899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.653875    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:11:51.667052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.667020    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:11:51.667465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.667225    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:11:51.672026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.671998    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"]
Nov 02 14:11:51.672155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.672140    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:11:51.754510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.754480    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef83544-2544-4d8b-9089-127607007b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929a74eb-f9gdb\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.754667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.754530    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef83544-2544-4d8b-9089-127607007b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929a74eb-f9gdb\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.755072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.754723    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vf4\" (UniqueName: \"kubernetes.io/projected/bef83544-2544-4d8b-9089-127607007b15-kube-api-access-f2vf4\") pod \"maas-api-revoke-sub-929a74eb-f9gdb\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.856129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.856097    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef83544-2544-4d8b-9089-127607007b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929a74eb-f9gdb\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.856278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.856143    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef83544-2544-4d8b-9089-127607007b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929a74eb-f9gdb\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.856278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.856228    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vf4\" (UniqueName: \"kubernetes.io/projected/bef83544-2544-4d8b-9089-127607007b15-kube-api-access-f2vf4\") pod \"maas-api-revoke-sub-929a74eb-f9gdb\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.857001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.856974    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef83544-2544-4d8b-9089-127607007b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929a74eb-f9gdb\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.859669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.859350    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef83544-2544-4d8b-9089-127607007b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929a74eb-f9gdb\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.889764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.889690    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vf4\" (UniqueName: \"kubernetes.io/projected/bef83544-2544-4d8b-9089-127607007b15-kube-api-access-f2vf4\") pod \"maas-api-revoke-sub-929a74eb-f9gdb\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:51.948353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:51.948319    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:52.263645 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:11:52.263610    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef83544_2544_4d8b_9089_127607007b15.slice/crio-7c20bad5195a386f23787b0eebbe5d1c9aed5a534dba4d9cdca0913bf36778cd WatchSource:0}: Error finding container 7c20bad5195a386f23787b0eebbe5d1c9aed5a534dba4d9cdca0913bf36778cd: Status 404 returned error can't find the container with id 7c20bad5195a386f23787b0eebbe5d1c9aed5a534dba4d9cdca0913bf36778cd
Nov 02 14:11:52.264639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:52.264611    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"]
Nov 02 14:11:52.635819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:52.635763    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb" event={"ID":"bef83544-2544-4d8b-9089-127607007b15","Type":"ContainerStarted","Data":"28ec3896d35f516cd134e7f9e8b94b49661801da494b2099f031524e25f42d82"}
Nov 02 14:11:52.636235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:52.635848    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb" event={"ID":"bef83544-2544-4d8b-9089-127607007b15","Type":"ContainerStarted","Data":"7c20bad5195a386f23787b0eebbe5d1c9aed5a534dba4d9cdca0913bf36778cd"}
Nov 02 14:11:52.685728 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:52.685682    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb" podStartSLOduration=1.685668255 podStartE2EDuration="1.685668255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:11:52.68110624 +0000 UTC m=+1703.432410677" watchObservedRunningTime="2026-11-02 14:11:52.685668255 +0000 UTC m=+1703.436972710"
Nov 02 14:11:53.645107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:53.645076    5034 generic.go:356] "Generic (PLEG): container finished" podID="bef83544-2544-4d8b-9089-127607007b15" containerID="28ec3896d35f516cd134e7f9e8b94b49661801da494b2099f031524e25f42d82" exitCode=0
Nov 02 14:11:53.645506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:53.645163    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb" event={"ID":"bef83544-2544-4d8b-9089-127607007b15","Type":"ContainerDied","Data":"28ec3896d35f516cd134e7f9e8b94b49661801da494b2099f031524e25f42d82"}
Nov 02 14:11:54.831563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:54.831539    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:55.003858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.003833    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bef83544-2544-4d8b-9089-127607007b15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef83544-2544-4d8b-9089-127607007b15-maas-api-xks-ca\") pod \"bef83544-2544-4d8b-9089-127607007b15\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") "
Nov 02 14:11:55.004021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.004007    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef83544-2544-4d8b-9089-127607007b15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef83544-2544-4d8b-9089-127607007b15-maas-api-openshift-ca\") pod \"bef83544-2544-4d8b-9089-127607007b15\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") "
Nov 02 14:11:55.004117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.004103    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef83544-2544-4d8b-9089-127607007b15-kube-api-access-f2vf4\" (UniqueName: \"kubernetes.io/projected/bef83544-2544-4d8b-9089-127607007b15-kube-api-access-f2vf4\") pod \"bef83544-2544-4d8b-9089-127607007b15\" (UID: \"bef83544-2544-4d8b-9089-127607007b15\") "
Nov 02 14:11:55.004430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.004401    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef83544-2544-4d8b-9089-127607007b15-maas-api-openshift-ca" pod "bef83544-2544-4d8b-9089-127607007b15" (UID: "bef83544-2544-4d8b-9089-127607007b15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:11:55.004597 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.004579    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bef83544-2544-4d8b-9089-127607007b15-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:55.006084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.006055    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef83544-2544-4d8b-9089-127607007b15-maas-api-xks-ca" pod "bef83544-2544-4d8b-9089-127607007b15" (UID: "bef83544-2544-4d8b-9089-127607007b15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:11:55.006607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.006587    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef83544-2544-4d8b-9089-127607007b15-kube-api-access-f2vf4" pod "bef83544-2544-4d8b-9089-127607007b15" (UID: "bef83544-2544-4d8b-9089-127607007b15"). InnerVolumeSpecName "kube-api-access-f2vf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:11:55.105541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.105513    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bef83544-2544-4d8b-9089-127607007b15-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:55.105541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.105542    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2vf4\" (UniqueName: \"kubernetes.io/projected/bef83544-2544-4d8b-9089-127607007b15-kube-api-access-f2vf4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:11:55.660081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.660002    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"
Nov 02 14:11:55.660081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.660038    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb" event={"ID":"bef83544-2544-4d8b-9089-127607007b15","Type":"ContainerDied","Data":"7c20bad5195a386f23787b0eebbe5d1c9aed5a534dba4d9cdca0913bf36778cd"}
Nov 02 14:11:55.660258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:11:55.660085    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c20bad5195a386f23787b0eebbe5d1c9aed5a534dba4d9cdca0913bf36778cd"
Nov 02 14:12:00.550332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:00.550295    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"]
Nov 02 14:12:00.550903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:00.550492    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt" podUID="3dc1483e-0366-4104-90ea-3c1cbaa0de70" containerName="payload-processing" containerID="cri-o://6d334c5c56d862a88e6f4228f9bc90929de952edb8b8f74d0f1651b9302d3daf" gracePeriod=30
Nov 02 14:12:00.605798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:00.587321    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"]
Nov 02 14:12:00.605798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:00.587592    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8" podUID="92e33df8-c051-420e-ba83-d86f6a253dc6" containerName="payload-pre-processing" containerID="cri-o://a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999" gracePeriod=30
Nov 02 14:12:00.699874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:00.699847    5034 generic.go:356] "Generic (PLEG): container finished" podID="3dc1483e-0366-4104-90ea-3c1cbaa0de70" containerID="6d334c5c56d862a88e6f4228f9bc90929de952edb8b8f74d0f1651b9302d3daf" exitCode=0
Nov 02 14:12:00.700012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:00.699925    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt" event={"ID":"3dc1483e-0366-4104-90ea-3c1cbaa0de70","Type":"ContainerDied","Data":"6d334c5c56d862a88e6f4228f9bc90929de952edb8b8f74d0f1651b9302d3daf"}
Nov 02 14:12:01.017459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.017430    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:12:01.023710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.023686    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dc1483e-0366-4104-90ea-3c1cbaa0de70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc1483e-0366-4104-90ea-3c1cbaa0de70-plugins-config-volume\") pod \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\" (UID: \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\") "
Nov 02 14:12:01.023911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.023899    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dc1483e-0366-4104-90ea-3c1cbaa0de70-kube-api-access-dzg8s\" (UniqueName: \"kubernetes.io/projected/3dc1483e-0366-4104-90ea-3c1cbaa0de70-kube-api-access-dzg8s\") pod \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\" (UID: \"3dc1483e-0366-4104-90ea-3c1cbaa0de70\") "
Nov 02 14:12:01.024930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.024902    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dc1483e-0366-4104-90ea-3c1cbaa0de70-plugins-config-volume" pod "3dc1483e-0366-4104-90ea-3c1cbaa0de70" (UID: "3dc1483e-0366-4104-90ea-3c1cbaa0de70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:01.032049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.032017    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc1483e-0366-4104-90ea-3c1cbaa0de70-kube-api-access-dzg8s" pod "3dc1483e-0366-4104-90ea-3c1cbaa0de70" (UID: "3dc1483e-0366-4104-90ea-3c1cbaa0de70"). InnerVolumeSpecName "kube-api-access-dzg8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:01.063157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.063136    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:12:01.124852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.124761    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e33df8-c051-420e-ba83-d86f6a253dc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e33df8-c051-420e-ba83-d86f6a253dc6-plugins-config-volume\") pod \"92e33df8-c051-420e-ba83-d86f6a253dc6\" (UID: \"92e33df8-c051-420e-ba83-d86f6a253dc6\") "
Nov 02 14:12:01.124852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.124826    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e33df8-c051-420e-ba83-d86f6a253dc6-kube-api-access-zj8vh\" (UniqueName: \"kubernetes.io/projected/92e33df8-c051-420e-ba83-d86f6a253dc6-kube-api-access-zj8vh\") pod \"92e33df8-c051-420e-ba83-d86f6a253dc6\" (UID: \"92e33df8-c051-420e-ba83-d86f6a253dc6\") "
Nov 02 14:12:01.125201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.125172    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e33df8-c051-420e-ba83-d86f6a253dc6-plugins-config-volume" pod "92e33df8-c051-420e-ba83-d86f6a253dc6" (UID: "92e33df8-c051-420e-ba83-d86f6a253dc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:01.125446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.125422    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e33df8-c051-420e-ba83-d86f6a253dc6-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:01.125446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.125445    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc1483e-0366-4104-90ea-3c1cbaa0de70-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:01.125622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.125454    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzg8s\" (UniqueName: \"kubernetes.io/projected/3dc1483e-0366-4104-90ea-3c1cbaa0de70-kube-api-access-dzg8s\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:01.127180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.127159    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e33df8-c051-420e-ba83-d86f6a253dc6-kube-api-access-zj8vh" pod "92e33df8-c051-420e-ba83-d86f6a253dc6" (UID: "92e33df8-c051-420e-ba83-d86f6a253dc6"). InnerVolumeSpecName "kube-api-access-zj8vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:01.227193 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.227162    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj8vh\" (UniqueName: \"kubernetes.io/projected/92e33df8-c051-420e-ba83-d86f6a253dc6-kube-api-access-zj8vh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:01.709108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.709018    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt" event={"ID":"3dc1483e-0366-4104-90ea-3c1cbaa0de70","Type":"ContainerDied","Data":"b31f4f820abaf03927cb7f111db50a72b7c599e273590463618c60d3fb0376fc"}
Nov 02 14:12:01.709108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.709085    5034 scope.go:122] "RemoveContainer" containerID="6d334c5c56d862a88e6f4228f9bc90929de952edb8b8f74d0f1651b9302d3daf"
Nov 02 14:12:01.709108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.709023    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"
Nov 02 14:12:01.710646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.710622    5034 generic.go:356] "Generic (PLEG): container finished" podID="92e33df8-c051-420e-ba83-d86f6a253dc6" containerID="a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999" exitCode=0
Nov 02 14:12:01.710813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.710794    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"
Nov 02 14:12:01.710922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.710802    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8" event={"ID":"92e33df8-c051-420e-ba83-d86f6a253dc6","Type":"ContainerDied","Data":"a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999"}
Nov 02 14:12:01.711020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.711003    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8" event={"ID":"92e33df8-c051-420e-ba83-d86f6a253dc6","Type":"ContainerDied","Data":"c08c6c184bc3b248025708bfa9a8d271d3f715cae35a8bc91b2202d49dc96b9a"}
Nov 02 14:12:01.719962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.719932    5034 scope.go:122] "RemoveContainer" containerID="a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999"
Nov 02 14:12:01.728982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.728964    5034 scope.go:122] "RemoveContainer" containerID="a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999"
Nov 02 14:12:01.729243 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:12:01.729224    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999\": container with ID starting with a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999 not found: ID does not exist" containerID="a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999"
Nov 02 14:12:01.729312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.729254    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999"} err="failed to get container status \"a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999\": rpc error: code = NotFound desc = could not find container \"a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999\": container with ID starting with a0faf70bf444ad42914361d7e750b98c3cf0e5f2b2197e04430ba12300965999 not found: ID does not exist"
Nov 02 14:12:01.767481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.767459    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"]
Nov 02 14:12:01.780935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.780915    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-794667-55b6d545b8-7nfdt"]
Nov 02 14:12:01.817312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.817279    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"]
Nov 02 14:12:01.822846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:01.822827    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-794667-6cdf7ff6dc-xk9d8"]
Nov 02 14:12:03.395650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:03.395617    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dc1483e-0366-4104-90ea-3c1cbaa0de70" path="/var/lib/kubelet/pods/3dc1483e-0366-4104-90ea-3c1cbaa0de70/volumes"
Nov 02 14:12:03.396138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:03.396123    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e33df8-c051-420e-ba83-d86f6a253dc6" path="/var/lib/kubelet/pods/92e33df8-c051-420e-ba83-d86f6a253dc6/volumes"
Nov 02 14:12:04.234415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.234386    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"]
Nov 02 14:12:04.235052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.235026    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3dc1483e-0366-4104-90ea-3c1cbaa0de70" containerName="payload-processing"
Nov 02 14:12:04.235052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.235052    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dc1483e-0366-4104-90ea-3c1cbaa0de70" containerName="payload-processing"
Nov 02 14:12:04.235228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.235097    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bef83544-2544-4d8b-9089-127607007b15" containerName="revoke-keys"
Nov 02 14:12:04.235228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.235106    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef83544-2544-4d8b-9089-127607007b15" containerName="revoke-keys"
Nov 02 14:12:04.235228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.235175    5034 cpu_manager.go:414] "containerMap: removing container" podUID="92e33df8-c051-420e-ba83-d86f6a253dc6" containerName="payload-pre-processing"
Nov 02 14:12:04.235228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.235184    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e33df8-c051-420e-ba83-d86f6a253dc6" containerName="payload-pre-processing"
Nov 02 14:12:04.235433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.235374    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e33df8-c051-420e-ba83-d86f6a253dc6" containerName="payload-pre-processing"
Nov 02 14:12:04.235433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.235392    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dc1483e-0366-4104-90ea-3c1cbaa0de70" containerName="payload-processing"
Nov 02 14:12:04.235433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.235408    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef83544-2544-4d8b-9089-127607007b15" containerName="revoke-keys"
Nov 02 14:12:04.240413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.240394    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.257208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.257187    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:12:04.264139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.264112    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b7289c8-6k2cd\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.264252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.264152    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdbf\" (UniqueName: \"kubernetes.io/projected/a9b54f09-2788-4fa9-8903-24c0aae57d4b-kube-api-access-qqdbf\") pod \"maas-api-revoke-sub-2b7289c8-6k2cd\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.264502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.264477    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b7289c8-6k2cd\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.270507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.270483    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:12:04.270648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.270542    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:12:04.270648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.270621    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:12:04.275294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.275266    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"]
Nov 02 14:12:04.366226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.366199    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b7289c8-6k2cd\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.366226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.366229    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdbf\" (UniqueName: \"kubernetes.io/projected/a9b54f09-2788-4fa9-8903-24c0aae57d4b-kube-api-access-qqdbf\") pod \"maas-api-revoke-sub-2b7289c8-6k2cd\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.366505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.366339    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b7289c8-6k2cd\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.366877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.366857    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b7289c8-6k2cd\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.368727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.368706    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b7289c8-6k2cd\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.390697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.390677    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdbf\" (UniqueName: \"kubernetes.io/projected/a9b54f09-2788-4fa9-8903-24c0aae57d4b-kube-api-access-qqdbf\") pod \"maas-api-revoke-sub-2b7289c8-6k2cd\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.555115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.555042    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:04.898195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:04.898161    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"]
Nov 02 14:12:05.239066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:05.239012    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"]
Nov 02 14:12:05.239707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:05.239671    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy" containerID="cri-o://50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae" gracePeriod=30
Nov 02 14:12:05.802960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:05.802923    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd" event={"ID":"a9b54f09-2788-4fa9-8903-24c0aae57d4b","Type":"ContainerStarted","Data":"fc587a8519d3f25d71457987834daebf0e379382cf6d8036e760a74d7db19433"}
Nov 02 14:12:05.803386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:05.802980    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd" event={"ID":"a9b54f09-2788-4fa9-8903-24c0aae57d4b","Type":"ContainerStarted","Data":"4d412d4f228a7213becaf3df9247f79174caaa6743ded70db8516c3791232947"}
Nov 02 14:12:05.852837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:05.852770    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd" podStartSLOduration=1.852755425 podStartE2EDuration="1.852755425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:05.851289571 +0000 UTC m=+1716.602594009" watchObservedRunningTime="2026-11-02 14:12:05.852755425 +0000 UTC m=+1716.604059856"
Nov 02 14:12:06.814824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:06.814763    5034 generic.go:356] "Generic (PLEG): container finished" podID="a9b54f09-2788-4fa9-8903-24c0aae57d4b" containerID="fc587a8519d3f25d71457987834daebf0e379382cf6d8036e760a74d7db19433" exitCode=0
Nov 02 14:12:06.815335 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:06.814844    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd" event={"ID":"a9b54f09-2788-4fa9-8903-24c0aae57d4b","Type":"ContainerDied","Data":"fc587a8519d3f25d71457987834daebf0e379382cf6d8036e760a74d7db19433"}
Nov 02 14:12:07.993140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:07.993116    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:08.031133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.031110    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-openshift-ca\") pod \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") "
Nov 02 14:12:08.031282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.031151    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b54f09-2788-4fa9-8903-24c0aae57d4b-kube-api-access-qqdbf\" (UniqueName: \"kubernetes.io/projected/a9b54f09-2788-4fa9-8903-24c0aae57d4b-kube-api-access-qqdbf\") pod \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") "
Nov 02 14:12:08.031282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.031239    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-xks-ca\") pod \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\" (UID: \"a9b54f09-2788-4fa9-8903-24c0aae57d4b\") "
Nov 02 14:12:08.031493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.031468    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-openshift-ca" pod "a9b54f09-2788-4fa9-8903-24c0aae57d4b" (UID: "a9b54f09-2788-4fa9-8903-24c0aae57d4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:08.031741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.031722    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:08.033183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.033159    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-xks-ca" pod "a9b54f09-2788-4fa9-8903-24c0aae57d4b" (UID: "a9b54f09-2788-4fa9-8903-24c0aae57d4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:08.033286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.033253    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b54f09-2788-4fa9-8903-24c0aae57d4b-kube-api-access-qqdbf" pod "a9b54f09-2788-4fa9-8903-24c0aae57d4b" (UID: "a9b54f09-2788-4fa9-8903-24c0aae57d4b"). InnerVolumeSpecName "kube-api-access-qqdbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:08.133366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.133285    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqdbf\" (UniqueName: \"kubernetes.io/projected/a9b54f09-2788-4fa9-8903-24c0aae57d4b-kube-api-access-qqdbf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:08.133366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.133322    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b54f09-2788-4fa9-8903-24c0aae57d4b-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:08.839627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.839596    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"
Nov 02 14:12:08.839847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.839641    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd" event={"ID":"a9b54f09-2788-4fa9-8903-24c0aae57d4b","Type":"ContainerDied","Data":"4d412d4f228a7213becaf3df9247f79174caaa6743ded70db8516c3791232947"}
Nov 02 14:12:08.839847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.839702    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d412d4f228a7213becaf3df9247f79174caaa6743ded70db8516c3791232947"
Nov 02 14:12:08.985799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.980474    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"]
Nov 02 14:12:08.985799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:08.980727    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy" containerID="cri-o://fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5" gracePeriod=30
Nov 02 14:12:10.577959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.577931    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:12:10.686110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.686079    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-data\") pod \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") "
Nov 02 14:12:10.686302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.686147    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-kube-api-access-rhfk9\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-kube-api-access-rhfk9\") pod \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") "
Nov 02 14:12:10.686302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.686179    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-credential-socket\") pod \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") "
Nov 02 14:12:10.686302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.686252    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-socket\") pod \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") "
Nov 02 14:12:10.686482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.686317    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-envoy\") pod \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") "
Nov 02 14:12:10.686482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.686371    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-token\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-token\") pod \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") "
Nov 02 14:12:10.686482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.686397    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-certs\") pod \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") "
Nov 02 14:12:10.686633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.686486    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istiod-ca-cert\") pod \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") "
Nov 02 14:12:10.686633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.686607    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-podinfo\") pod \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\" (UID: \"b22ed9db-6d6c-4575-bff2-881a07c7aa04\") "
Nov 02 14:12:10.687512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.687484    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-socket" pod "b22ed9db-6d6c-4575-bff2-881a07c7aa04" (UID: "b22ed9db-6d6c-4575-bff2-881a07c7aa04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:10.687683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.687664    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-data" pod "b22ed9db-6d6c-4575-bff2-881a07c7aa04" (UID: "b22ed9db-6d6c-4575-bff2-881a07c7aa04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:10.689772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.689741    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-podinfo" pod "b22ed9db-6d6c-4575-bff2-881a07c7aa04" (UID: "b22ed9db-6d6c-4575-bff2-881a07c7aa04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:12:10.689772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.689753    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-kube-api-access-rhfk9" pod "b22ed9db-6d6c-4575-bff2-881a07c7aa04" (UID: "b22ed9db-6d6c-4575-bff2-881a07c7aa04"). InnerVolumeSpecName "kube-api-access-rhfk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:10.690012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.689991    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-credential-socket" pod "b22ed9db-6d6c-4575-bff2-881a07c7aa04" (UID: "b22ed9db-6d6c-4575-bff2-881a07c7aa04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:10.690169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.690148    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-certs" pod "b22ed9db-6d6c-4575-bff2-881a07c7aa04" (UID: "b22ed9db-6d6c-4575-bff2-881a07c7aa04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:10.691929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.691904    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-token" pod "b22ed9db-6d6c-4575-bff2-881a07c7aa04" (UID: "b22ed9db-6d6c-4575-bff2-881a07c7aa04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:10.692231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.692210    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istiod-ca-cert" pod "b22ed9db-6d6c-4575-bff2-881a07c7aa04" (UID: "b22ed9db-6d6c-4575-bff2-881a07c7aa04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:10.692547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.692526    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-envoy" pod "b22ed9db-6d6c-4575-bff2-881a07c7aa04" (UID: "b22ed9db-6d6c-4575-bff2-881a07c7aa04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:10.788535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.788503    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:10.788860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.788844    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:10.789041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.789029    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:10.789174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.789161    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhfk9\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-kube-api-access-rhfk9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:10.789279 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.789267    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:10.789408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.789396    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:10.789500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.789489    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:10.789584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.789566    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22ed9db-6d6c-4575-bff2-881a07c7aa04-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:10.789685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.789674    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22ed9db-6d6c-4575-bff2-881a07c7aa04-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:10.856349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.856255    5034 generic.go:356] "Generic (PLEG): container finished" podID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerID="50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae" exitCode=0
Nov 02 14:12:10.856349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.856307    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"
Nov 02 14:12:10.856576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.856362    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" event={"ID":"b22ed9db-6d6c-4575-bff2-881a07c7aa04","Type":"ContainerDied","Data":"50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae"}
Nov 02 14:12:10.856576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.856424    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5" event={"ID":"b22ed9db-6d6c-4575-bff2-881a07c7aa04","Type":"ContainerDied","Data":"77cf48fd0ea2955a17dd2babe243f54bd8105853f336efd0180582d23cb40ffb"}
Nov 02 14:12:10.856576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.856452    5034 scope.go:122] "RemoveContainer" containerID="50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae"
Nov 02 14:12:10.871420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.871396    5034 scope.go:122] "RemoveContainer" containerID="eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc"
Nov 02 14:12:10.987532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.983468    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"]
Nov 02 14:12:10.987532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.985147    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-794667-openshift-default-7f5ddd9b77-2ccq5"]
Nov 02 14:12:10.987532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.986906    5034 scope.go:122] "RemoveContainer" containerID="50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae"
Nov 02 14:12:10.987532 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:12:10.987419    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae\": container with ID starting with 50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae not found: ID does not exist" containerID="50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae"
Nov 02 14:12:10.987532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.987455    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae"} err="failed to get container status \"50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae\": rpc error: code = NotFound desc = could not find container \"50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae\": container with ID starting with 50b7cae9d4b49690c51e1daf21d5c2ea9778d31c38e331fbf898604ea39caeae not found: ID does not exist"
Nov 02 14:12:10.987532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.987479    5034 scope.go:122] "RemoveContainer" containerID="eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc"
Nov 02 14:12:10.988224 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:12:10.988195    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc\": container with ID starting with eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc not found: ID does not exist" containerID="eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc"
Nov 02 14:12:10.988324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:10.988233    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc"} err="failed to get container status \"eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc\": rpc error: code = NotFound desc = could not find container \"eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc\": container with ID starting with eedc4ef6fbf1471c66b88d7ac11f171397c54c01ce4cd45055b76afcdca064dc not found: ID does not exist"
Nov 02 14:12:11.396485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:11.396448    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" path="/var/lib/kubelet/pods/b22ed9db-6d6c-4575-bff2-881a07c7aa04/volumes"
Nov 02 14:12:12.511602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.511569    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"]
Nov 02 14:12:12.512652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.512624    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy"
Nov 02 14:12:12.512652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.512648    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy"
Nov 02 14:12:12.512906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.512664    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a9b54f09-2788-4fa9-8903-24c0aae57d4b" containerName="revoke-keys"
Nov 02 14:12:12.512906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.512671    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b54f09-2788-4fa9-8903-24c0aae57d4b" containerName="revoke-keys"
Nov 02 14:12:12.513023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.512947    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy"
Nov 02 14:12:12.513023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.512985    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b54f09-2788-4fa9-8903-24c0aae57d4b" containerName="revoke-keys"
Nov 02 14:12:12.518407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.518380    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.570121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.570092    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"]
Nov 02 14:12:12.626426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.626392    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.626582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.626428    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.626582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.626528    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.626582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.626554    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.626709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.626689    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.626832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.626729    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.626832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.626755    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.626966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.626950    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmm54\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-kube-api-access-tmm54\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.627006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.626998    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728310    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728349    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728366    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728484    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmm54\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-kube-api-access-tmm54\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728540    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728640    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728665    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728700    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728726    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728762    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.728923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.728836    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-credential-socket\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.729486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.729170    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-data\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.729486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.729241    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-certs\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.729612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.729502    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istiod-ca-cert\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.730923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.730902    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-envoy\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.731146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.731126    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-podinfo\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.754527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.754506    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-token\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.754897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.754868    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmm54\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-kube-api-access-tmm54\") pod \"e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") " pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.834190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.834113    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:12.838992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.838947    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"]
Nov 02 14:12:12.840395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.839950    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy"
Nov 02 14:12:12.840395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.839972    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy"
Nov 02 14:12:12.840395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.840142    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22ed9db-6d6c-4575-bff2-881a07c7aa04" containerName="istio-proxy"
Nov 02 14:12:12.853238 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.853214    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:12.870515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.870473    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c07f7e-openshift-default-dockercfg-4p5g5\""
Nov 02 14:12:12.879933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.879909    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"]
Nov 02 14:12:12.933628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.933408    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69662c7-cd64-4242-84fb-bd6deea012be-istio-podinfo\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:12.933628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.933468    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-istio-token\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:12.933628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.933508    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69662c7-cd64-4242-84fb-bd6deea012be-istiod-ca-cert\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:12.937152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.933811    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-data\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:12.937152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.933916    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-socket\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:12.937152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.933993    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj49m\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-kube-api-access-mj49m\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:12.937152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.934151    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-credential-socket\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:12.937152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.934248    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-certs\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:12.937152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:12.934368    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-envoy\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.048994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.048347    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69662c7-cd64-4242-84fb-bd6deea012be-istio-podinfo\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.048994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.048403    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-istio-token\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.048994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.048435    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69662c7-cd64-4242-84fb-bd6deea012be-istiod-ca-cert\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.048994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.048552    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-data\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.048994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.048627    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-socket\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.048994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.048675    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj49m\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-kube-api-access-mj49m\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.048994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.048807    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-credential-socket\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.048994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.048871    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-certs\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.050172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.049628    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-envoy\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.053325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.052913    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-envoy\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.057731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.055999    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69662c7-cd64-4242-84fb-bd6deea012be-istio-podinfo\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.057731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.056494    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-credential-socket\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.057731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.056738    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-certs\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.057731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.056975    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-data\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.057731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.057359    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69662c7-cd64-4242-84fb-bd6deea012be-istiod-ca-cert\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.059831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.058693    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-socket\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.086396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.086302    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-istio-token\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.088063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.088006    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj49m\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-kube-api-access-mj49m\") pod \"e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") " pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.203054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.203008    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"]
Nov 02 14:12:13.203525 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:12:13.203495    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d79516_f3b6_46ed_b3bc_e43bfc44465f.slice/crio-233cc2431bb70bf76d837251e723c7e8a0b3df0ce8a13566ad5f625f962b4563 WatchSource:0}: Error finding container 233cc2431bb70bf76d837251e723c7e8a0b3df0ce8a13566ad5f625f962b4563: Status 404 returned error can't find the container with id 233cc2431bb70bf76d837251e723c7e8a0b3df0ce8a13566ad5f625f962b4563
Nov 02 14:12:13.206477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.206440    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:12:13.206602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.206521    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:12:13.206602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.206565    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:12:13.211475 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.211453    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:13.542268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.542238    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"]
Nov 02 14:12:13.542705 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:12:13.542284    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69662c7_cd64_4242_84fb_bd6deea012be.slice/crio-20918ef7a658a2017d45f4a152c95cebe7b9dcb8b8ac79c006e76afa6ac9569b WatchSource:0}: Error finding container 20918ef7a658a2017d45f4a152c95cebe7b9dcb8b8ac79c006e76afa6ac9569b: Status 404 returned error can't find the container with id 20918ef7a658a2017d45f4a152c95cebe7b9dcb8b8ac79c006e76afa6ac9569b
Nov 02 14:12:13.545932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.545896    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:12:13.546022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.545979    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:12:13.546072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.546022    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:12:13.896168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.896065    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" event={"ID":"44d79516-f3b6-46ed-b3bc-e43bfc44465f","Type":"ContainerStarted","Data":"65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4"}
Nov 02 14:12:13.896168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.896133    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" event={"ID":"44d79516-f3b6-46ed-b3bc-e43bfc44465f","Type":"ContainerStarted","Data":"233cc2431bb70bf76d837251e723c7e8a0b3df0ce8a13566ad5f625f962b4563"}
Nov 02 14:12:13.898005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.897967    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" event={"ID":"b69662c7-cd64-4242-84fb-bd6deea012be","Type":"ContainerStarted","Data":"c59ae6cb7f477809dfeb513f2aeaf5ac4b80849419bae81064b12d394e1c6c40"}
Nov 02 14:12:13.898105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.898029    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" event={"ID":"b69662c7-cd64-4242-84fb-bd6deea012be","Type":"ContainerStarted","Data":"20918ef7a658a2017d45f4a152c95cebe7b9dcb8b8ac79c006e76afa6ac9569b"}
Nov 02 14:12:13.973495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:13.973442    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" podStartSLOduration=1.9734277740000001 podStartE2EDuration="1.973427774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:13.968794504 +0000 UTC m=+1724.720098933" watchObservedRunningTime="2026-11-02 14:12:13.973427774 +0000 UTC m=+1724.724732211"
Nov 02 14:12:14.029995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.029945    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" podStartSLOduration=2.029929887 podStartE2EDuration="2.029929887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:14.028006332 +0000 UTC m=+1724.779310769" watchObservedRunningTime="2026-11-02 14:12:14.029929887 +0000 UTC m=+1724.781234399"
Nov 02 14:12:14.214815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.212247    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:14.230270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.228854    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:14.324802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.308197    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:12:14.367729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.367695    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-data\") pod \"8f666d7f-79d0-4556-b859-62fd24beb2a0\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") "
Nov 02 14:12:14.367920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.367748    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-socket\") pod \"8f666d7f-79d0-4556-b859-62fd24beb2a0\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") "
Nov 02 14:12:14.367920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.367821    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-kube-api-access-n2fsr\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-kube-api-access-n2fsr\") pod \"8f666d7f-79d0-4556-b859-62fd24beb2a0\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") "
Nov 02 14:12:14.367920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.367870    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-credential-socket\") pod \"8f666d7f-79d0-4556-b859-62fd24beb2a0\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") "
Nov 02 14:12:14.368114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368023    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-data" pod "8f666d7f-79d0-4556-b859-62fd24beb2a0" (UID: "8f666d7f-79d0-4556-b859-62fd24beb2a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:14.368114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368037    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-socket" pod "8f666d7f-79d0-4556-b859-62fd24beb2a0" (UID: "8f666d7f-79d0-4556-b859-62fd24beb2a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:14.368114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368050    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-envoy\") pod \"8f666d7f-79d0-4556-b859-62fd24beb2a0\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") "
Nov 02 14:12:14.368114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368105    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-certs\") pod \"8f666d7f-79d0-4556-b859-62fd24beb2a0\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") "
Nov 02 14:12:14.368320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368196    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-credential-socket" pod "8f666d7f-79d0-4556-b859-62fd24beb2a0" (UID: "8f666d7f-79d0-4556-b859-62fd24beb2a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:14.368320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368274    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f666d7f-79d0-4556-b859-62fd24beb2a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f666d7f-79d0-4556-b859-62fd24beb2a0-istiod-ca-cert\") pod \"8f666d7f-79d0-4556-b859-62fd24beb2a0\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") "
Nov 02 14:12:14.368320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368303    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-podinfo\") pod \"8f666d7f-79d0-4556-b859-62fd24beb2a0\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") "
Nov 02 14:12:14.368544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368343    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-token\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-token\") pod \"8f666d7f-79d0-4556-b859-62fd24beb2a0\" (UID: \"8f666d7f-79d0-4556-b859-62fd24beb2a0\") "
Nov 02 14:12:14.368544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368358    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-certs" pod "8f666d7f-79d0-4556-b859-62fd24beb2a0" (UID: "8f666d7f-79d0-4556-b859-62fd24beb2a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:14.368657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.368608    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f666d7f-79d0-4556-b859-62fd24beb2a0-istiod-ca-cert" pod "8f666d7f-79d0-4556-b859-62fd24beb2a0" (UID: "8f666d7f-79d0-4556-b859-62fd24beb2a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:14.407022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.384938    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:14.407022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.384973    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f666d7f-79d0-4556-b859-62fd24beb2a0-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:14.407022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.384986    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:14.407022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.384999    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:14.407022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.385012    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:14.407022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.394971    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-envoy" pod "8f666d7f-79d0-4556-b859-62fd24beb2a0" (UID: "8f666d7f-79d0-4556-b859-62fd24beb2a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:12:14.410044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.409983    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-kube-api-access-n2fsr" pod "8f666d7f-79d0-4556-b859-62fd24beb2a0" (UID: "8f666d7f-79d0-4556-b859-62fd24beb2a0"). InnerVolumeSpecName "kube-api-access-n2fsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:14.410180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.410077    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-podinfo" pod "8f666d7f-79d0-4556-b859-62fd24beb2a0" (UID: "8f666d7f-79d0-4556-b859-62fd24beb2a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:12:14.410595 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.410556    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-token" pod "8f666d7f-79d0-4556-b859-62fd24beb2a0" (UID: "8f666d7f-79d0-4556-b859-62fd24beb2a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:14.487811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.487742    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:14.488086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.488066    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:14.488301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.488285    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:14.488453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.488418    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2fsr\" (UniqueName: \"kubernetes.io/projected/8f666d7f-79d0-4556-b859-62fd24beb2a0-kube-api-access-n2fsr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:14.834304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.834269    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:14.835592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.835561    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 14:12:14.835731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.835610    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 14:12:14.911887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.911846    5034 generic.go:356] "Generic (PLEG): container finished" podID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerID="fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5" exitCode=0
Nov 02 14:12:14.912093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.911901    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"
Nov 02 14:12:14.912093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.911920    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" event={"ID":"8f666d7f-79d0-4556-b859-62fd24beb2a0","Type":"ContainerDied","Data":"fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5"}
Nov 02 14:12:14.912093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.911987    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp" event={"ID":"8f666d7f-79d0-4556-b859-62fd24beb2a0","Type":"ContainerDied","Data":"b0e7c0e4074d2855cd6d7fcd8e730c43c2ecc4b6c92bd3a32be78bfd31ab56de"}
Nov 02 14:12:14.912093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.912021    5034 scope.go:122] "RemoveContainer" containerID="fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5"
Nov 02 14:12:14.913270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.913235    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:14.914669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.914649    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:12:14.924242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.924225    5034 scope.go:122] "RemoveContainer" containerID="d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328"
Nov 02 14:12:14.934093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.934070    5034 scope.go:122] "RemoveContainer" containerID="fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5"
Nov 02 14:12:14.934357 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:12:14.934340    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5\": container with ID starting with fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5 not found: ID does not exist" containerID="fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5"
Nov 02 14:12:14.934398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.934365    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5"} err="failed to get container status \"fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5\": rpc error: code = NotFound desc = could not find container \"fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5\": container with ID starting with fbb89f8752dd37efd7e63500bb44afadde51e9e89669bb777d20999a01f345c5 not found: ID does not exist"
Nov 02 14:12:14.934398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.934384    5034 scope.go:122] "RemoveContainer" containerID="d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328"
Nov 02 14:12:14.934626 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:12:14.934608    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328\": container with ID starting with d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328 not found: ID does not exist" containerID="d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328"
Nov 02 14:12:14.934672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:14.934632    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328"} err="failed to get container status \"d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328\": rpc error: code = NotFound desc = could not find container \"d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328\": container with ID starting with d0e6c29e08df67bf766490c63ab11e52705eb1ae43cf19b5e48104558d848328 not found: ID does not exist"
Nov 02 14:12:15.044806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:15.044691    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"]
Nov 02 14:12:15.050203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:15.050167    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-370905-openshift-default-78c68564b5-td8wp"]
Nov 02 14:12:15.408833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:15.408730    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" path="/var/lib/kubelet/pods/8f666d7f-79d0-4556-b859-62fd24beb2a0/volumes"
Nov 02 14:12:15.835442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:15.835403    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 14:12:15.835879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:15.835482    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 14:12:16.229171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.229097    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"]
Nov 02 14:12:16.229723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.229707    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy"
Nov 02 14:12:16.229723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.229722    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy"
Nov 02 14:12:16.229897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.229886    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy"
Nov 02 14:12:16.235133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.235116    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.254647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.254618    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"]
Nov 02 14:12:16.264184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.264159    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:12:16.264305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.264250    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:12:16.264384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.264370    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:12:16.265925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.265909    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:12:16.328289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.328253    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdecd625-5k2jh\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.328462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.328298    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj8ds\" (UniqueName: \"kubernetes.io/projected/2e6c3549-effa-43d5-b052-28f1d8800600-kube-api-access-gj8ds\") pod \"maas-api-revoke-sub-fdecd625-5k2jh\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.328462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.328375    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdecd625-5k2jh\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.429334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.429300    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdecd625-5k2jh\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.429513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.429469    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdecd625-5k2jh\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.429513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.429494    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj8ds\" (UniqueName: \"kubernetes.io/projected/2e6c3549-effa-43d5-b052-28f1d8800600-kube-api-access-gj8ds\") pod \"maas-api-revoke-sub-fdecd625-5k2jh\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.430136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.430108    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdecd625-5k2jh\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.431687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.431663    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdecd625-5k2jh\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.456982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.456958    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj8ds\" (UniqueName: \"kubernetes.io/projected/2e6c3549-effa-43d5-b052-28f1d8800600-kube-api-access-gj8ds\") pod \"maas-api-revoke-sub-fdecd625-5k2jh\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.549641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.549613    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:16.834674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.834587    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 14:12:16.834674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.834647    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 14:12:16.910579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.910549    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"]
Nov 02 14:12:16.934469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:16.934442    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh" event={"ID":"2e6c3549-effa-43d5-b052-28f1d8800600","Type":"ContainerStarted","Data":"fe5e0b56c351c4807ff177ee37edf59b99f2c80e9a6b093c35b7b2348cede687"}
Nov 02 14:12:17.505969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.505932    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"]
Nov 02 14:12:17.506952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.506928    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy"
Nov 02 14:12:17.507042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.506954    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy"
Nov 02 14:12:17.507169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.507155    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f666d7f-79d0-4556-b859-62fd24beb2a0" containerName="istio-proxy"
Nov 02 14:12:17.511348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.511326    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.534442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.534420    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"]
Nov 02 14:12:17.651479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.651451    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ql5b\" (UniqueName: \"kubernetes.io/projected/5cc0c34a-eafe-4374-a00c-29973601d7ed-kube-api-access-5ql5b\") pod \"maas-api-revoke-sub-f238028e-5xqvj\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.651649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.651499    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f238028e-5xqvj\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.651695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.651647    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f238028e-5xqvj\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.753181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.753148    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f238028e-5xqvj\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.753350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.753264    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f238028e-5xqvj\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.753461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.753443    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ql5b\" (UniqueName: \"kubernetes.io/projected/5cc0c34a-eafe-4374-a00c-29973601d7ed-kube-api-access-5ql5b\") pod \"maas-api-revoke-sub-f238028e-5xqvj\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.753974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.753942    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f238028e-5xqvj\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.755483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.755459    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f238028e-5xqvj\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.799854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.792268    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ql5b\" (UniqueName: \"kubernetes.io/projected/5cc0c34a-eafe-4374-a00c-29973601d7ed-kube-api-access-5ql5b\") pod \"maas-api-revoke-sub-f238028e-5xqvj\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.826635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.826610    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:17.835256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.835234    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 14:12:17.835381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.835275    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 14:12:17.950951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:17.950916    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh" event={"ID":"2e6c3549-effa-43d5-b052-28f1d8800600","Type":"ContainerStarted","Data":"c6fd394deabebabbf3930ee381f683264d92967f54673040b5342f9310e9cf13"}
Nov 02 14:12:18.054571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:18.054487    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh" podStartSLOduration=2.054475405 podStartE2EDuration="2.054475405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:18.052503535 +0000 UTC m=+1728.803807973" watchObservedRunningTime="2026-11-02 14:12:18.054475405 +0000 UTC m=+1728.805779843"
Nov 02 14:12:18.207510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:18.207481    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"]
Nov 02 14:12:18.211535 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:12:18.211502    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc0c34a_eafe_4374_a00c_29973601d7ed.slice/crio-bf65c960dc7b3e961fa8e2b62a77ac13edbc076bf648b5f063857ac81597dc62 WatchSource:0}: Error finding container bf65c960dc7b3e961fa8e2b62a77ac13edbc076bf648b5f063857ac81597dc62: Status 404 returned error can't find the container with id bf65c960dc7b3e961fa8e2b62a77ac13edbc076bf648b5f063857ac81597dc62
Nov 02 14:12:18.575390 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:12:18.575315    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:12:18.835145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:18.835029    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 14:12:18.835145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:18.835090    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 14:12:18.959143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:18.959106    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj" event={"ID":"5cc0c34a-eafe-4374-a00c-29973601d7ed","Type":"ContainerStarted","Data":"5964ca171127db3aecb9b96cc4cecfd097910ec44b7cf29ff5fb8eebf9ecc067"}
Nov 02 14:12:18.959616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:18.959165    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj" event={"ID":"5cc0c34a-eafe-4374-a00c-29973601d7ed","Type":"ContainerStarted","Data":"bf65c960dc7b3e961fa8e2b62a77ac13edbc076bf648b5f063857ac81597dc62"}
Nov 02 14:12:18.960984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:18.960957    5034 generic.go:356] "Generic (PLEG): container finished" podID="2e6c3549-effa-43d5-b052-28f1d8800600" containerID="c6fd394deabebabbf3930ee381f683264d92967f54673040b5342f9310e9cf13" exitCode=0
Nov 02 14:12:18.961101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:18.961036    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh" event={"ID":"2e6c3549-effa-43d5-b052-28f1d8800600","Type":"ContainerDied","Data":"c6fd394deabebabbf3930ee381f683264d92967f54673040b5342f9310e9cf13"}
Nov 02 14:12:19.012509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:19.012456    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj" podStartSLOduration=2.012437075 podStartE2EDuration="2.012437075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:19.007083055 +0000 UTC m=+1729.758387493" watchObservedRunningTime="2026-11-02 14:12:19.012437075 +0000 UTC m=+1729.763741516"
Nov 02 14:12:19.834623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:19.834590    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body=
Nov 02 14:12:19.834865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:19.834651    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused"
Nov 02 14:12:19.972106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:19.972069    5034 generic.go:356] "Generic (PLEG): container finished" podID="5cc0c34a-eafe-4374-a00c-29973601d7ed" containerID="5964ca171127db3aecb9b96cc4cecfd097910ec44b7cf29ff5fb8eebf9ecc067" exitCode=0
Nov 02 14:12:19.972593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:19.972126    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj" event={"ID":"5cc0c34a-eafe-4374-a00c-29973601d7ed","Type":"ContainerDied","Data":"5964ca171127db3aecb9b96cc4cecfd097910ec44b7cf29ff5fb8eebf9ecc067"}
Nov 02 14:12:20.167648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.167619    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:20.224058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.224026    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6c3549-effa-43d5-b052-28f1d8800600-kube-api-access-gj8ds\" (UniqueName: \"kubernetes.io/projected/2e6c3549-effa-43d5-b052-28f1d8800600-kube-api-access-gj8ds\") pod \"2e6c3549-effa-43d5-b052-28f1d8800600\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") "
Nov 02 14:12:20.224262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.224240    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-xks-ca\") pod \"2e6c3549-effa-43d5-b052-28f1d8800600\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") "
Nov 02 14:12:20.224406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.224387    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-openshift-ca\") pod \"2e6c3549-effa-43d5-b052-28f1d8800600\" (UID: \"2e6c3549-effa-43d5-b052-28f1d8800600\") "
Nov 02 14:12:20.224949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.224925    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-openshift-ca" pod "2e6c3549-effa-43d5-b052-28f1d8800600" (UID: "2e6c3549-effa-43d5-b052-28f1d8800600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:20.229833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.227086    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6c3549-effa-43d5-b052-28f1d8800600-kube-api-access-gj8ds" pod "2e6c3549-effa-43d5-b052-28f1d8800600" (UID: "2e6c3549-effa-43d5-b052-28f1d8800600"). InnerVolumeSpecName "kube-api-access-gj8ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:20.229833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.227303    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-xks-ca" pod "2e6c3549-effa-43d5-b052-28f1d8800600" (UID: "2e6c3549-effa-43d5-b052-28f1d8800600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:20.325845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.325773    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj8ds\" (UniqueName: \"kubernetes.io/projected/2e6c3549-effa-43d5-b052-28f1d8800600-kube-api-access-gj8ds\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:20.325845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.325824    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:20.325845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.325834    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6c3549-effa-43d5-b052-28f1d8800600-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:20.985810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.985750    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"
Nov 02 14:12:20.985810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.985791    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh" event={"ID":"2e6c3549-effa-43d5-b052-28f1d8800600","Type":"ContainerDied","Data":"fe5e0b56c351c4807ff177ee37edf59b99f2c80e9a6b093c35b7b2348cede687"}
Nov 02 14:12:20.986361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:20.985853    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe5e0b56c351c4807ff177ee37edf59b99f2c80e9a6b093c35b7b2348cede687"
Nov 02 14:12:21.205792 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.205750    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:21.237868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.237793    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-openshift-ca\") pod \"5cc0c34a-eafe-4374-a00c-29973601d7ed\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") "
Nov 02 14:12:21.237868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.237857    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc0c34a-eafe-4374-a00c-29973601d7ed-kube-api-access-5ql5b\" (UniqueName: \"kubernetes.io/projected/5cc0c34a-eafe-4374-a00c-29973601d7ed-kube-api-access-5ql5b\") pod \"5cc0c34a-eafe-4374-a00c-29973601d7ed\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") "
Nov 02 14:12:21.238084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.237913    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-xks-ca\") pod \"5cc0c34a-eafe-4374-a00c-29973601d7ed\" (UID: \"5cc0c34a-eafe-4374-a00c-29973601d7ed\") "
Nov 02 14:12:21.238084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.238052    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-openshift-ca" pod "5cc0c34a-eafe-4374-a00c-29973601d7ed" (UID: "5cc0c34a-eafe-4374-a00c-29973601d7ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:21.238517 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.238479    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:21.240104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.240084    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-xks-ca" pod "5cc0c34a-eafe-4374-a00c-29973601d7ed" (UID: "5cc0c34a-eafe-4374-a00c-29973601d7ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:21.240195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.240080    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc0c34a-eafe-4374-a00c-29973601d7ed-kube-api-access-5ql5b" pod "5cc0c34a-eafe-4374-a00c-29973601d7ed" (UID: "5cc0c34a-eafe-4374-a00c-29973601d7ed"). InnerVolumeSpecName "kube-api-access-5ql5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:21.339869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.339839    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ql5b\" (UniqueName: \"kubernetes.io/projected/5cc0c34a-eafe-4374-a00c-29973601d7ed-kube-api-access-5ql5b\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:21.339869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.339871    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cc0c34a-eafe-4374-a00c-29973601d7ed-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:21.376118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.376085    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:21.376813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.376765    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:21.377975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.377956    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:12:21.995213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.995185    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"
Nov 02 14:12:21.995690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.995187    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj" event={"ID":"5cc0c34a-eafe-4374-a00c-29973601d7ed","Type":"ContainerDied","Data":"bf65c960dc7b3e961fa8e2b62a77ac13edbc076bf648b5f063857ac81597dc62"}
Nov 02 14:12:21.995690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:21.995355    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf65c960dc7b3e961fa8e2b62a77ac13edbc076bf648b5f063857ac81597dc62"
Nov 02 14:12:26.641352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.641317    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"]
Nov 02 14:12:26.642144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.642123    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2e6c3549-effa-43d5-b052-28f1d8800600" containerName="revoke-keys"
Nov 02 14:12:26.642253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.642145    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6c3549-effa-43d5-b052-28f1d8800600" containerName="revoke-keys"
Nov 02 14:12:26.642253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.642165    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5cc0c34a-eafe-4374-a00c-29973601d7ed" containerName="revoke-keys"
Nov 02 14:12:26.642253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.642173    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc0c34a-eafe-4374-a00c-29973601d7ed" containerName="revoke-keys"
Nov 02 14:12:26.642408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.642358    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6c3549-effa-43d5-b052-28f1d8800600" containerName="revoke-keys"
Nov 02 14:12:26.642408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.642377    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc0c34a-eafe-4374-a00c-29973601d7ed" containerName="revoke-keys"
Nov 02 14:12:26.645904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.645884    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.662737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.662714    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:12:26.666398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.666369    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"]
Nov 02 14:12:26.676831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.675735    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:12:26.676831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.675897    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:12:26.676831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.676143    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:12:26.744206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.744175    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7g78\" (UniqueName: \"kubernetes.io/projected/8731eb60-f299-4c3c-8224-1b4a5f57a94c-kube-api-access-r7g78\") pod \"maas-api-revoke-sub-9056b5d3-rbs2s\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.744358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.744215    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9056b5d3-rbs2s\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.744401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.744379    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9056b5d3-rbs2s\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.846489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.846447    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7g78\" (UniqueName: \"kubernetes.io/projected/8731eb60-f299-4c3c-8224-1b4a5f57a94c-kube-api-access-r7g78\") pod \"maas-api-revoke-sub-9056b5d3-rbs2s\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.846489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.846487    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9056b5d3-rbs2s\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.846751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.846640    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9056b5d3-rbs2s\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.847383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.847360    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9056b5d3-rbs2s\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.849302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.849278    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9056b5d3-rbs2s\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.871188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.871164    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7g78\" (UniqueName: \"kubernetes.io/projected/8731eb60-f299-4c3c-8224-1b4a5f57a94c-kube-api-access-r7g78\") pod \"maas-api-revoke-sub-9056b5d3-rbs2s\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:26.960298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:26.960237    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:27.279001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:27.278975    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"]
Nov 02 14:12:27.281382 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:12:27.281347    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8731eb60_f299_4c3c_8224_1b4a5f57a94c.slice/crio-2e86a5c75b4eef720ad50c56fcda9d354d8d74324721405689565e11ebc1dbd5 WatchSource:0}: Error finding container 2e86a5c75b4eef720ad50c56fcda9d354d8d74324721405689565e11ebc1dbd5: Status 404 returned error can't find the container with id 2e86a5c75b4eef720ad50c56fcda9d354d8d74324721405689565e11ebc1dbd5
Nov 02 14:12:27.909637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:27.909547    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"]
Nov 02 14:12:27.913865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:27.913842    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:27.933378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:27.933355    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"]
Nov 02 14:12:27.961015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:27.960988    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a034a743-d5n6v\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:27.961209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:27.961189    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a034a743-d5n6v\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:27.961254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:27.961224    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6thv\" (UniqueName: \"kubernetes.io/projected/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-kube-api-access-t6thv\") pod \"maas-api-revoke-sub-a034a743-d5n6v\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:28.050194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.050156    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s" event={"ID":"8731eb60-f299-4c3c-8224-1b4a5f57a94c","Type":"ContainerStarted","Data":"7a804a2b2e0197615738518b496f195194310d446275fc099152adcd2b2ae8d2"}
Nov 02 14:12:28.050340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.050205    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s" event={"ID":"8731eb60-f299-4c3c-8224-1b4a5f57a94c","Type":"ContainerStarted","Data":"2e86a5c75b4eef720ad50c56fcda9d354d8d74324721405689565e11ebc1dbd5"}
Nov 02 14:12:28.062954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.062930    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a034a743-d5n6v\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:28.063144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.063125    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a034a743-d5n6v\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:28.063211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.063160    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6thv\" (UniqueName: \"kubernetes.io/projected/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-kube-api-access-t6thv\") pod \"maas-api-revoke-sub-a034a743-d5n6v\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:28.063755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.063733    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a034a743-d5n6v\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:28.065350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.065327    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a034a743-d5n6v\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:28.098170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.098139    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6thv\" (UniqueName: \"kubernetes.io/projected/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-kube-api-access-t6thv\") pod \"maas-api-revoke-sub-a034a743-d5n6v\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:28.107258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.107203    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s" podStartSLOduration=2.107185517 podStartE2EDuration="2.107185517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:28.102443305 +0000 UTC m=+1738.853747745" watchObservedRunningTime="2026-11-02 14:12:28.107185517 +0000 UTC m=+1738.858489956"
Nov 02 14:12:28.230291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.230263    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:28.538657 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:12:28.538620    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0e18b7_26ee_46a7_aaf2_d16f46a25f56.slice/crio-769887432b11a94ada92d2d119ffc19d3df26b1fc018ea5c391f148a92b02043 WatchSource:0}: Error finding container 769887432b11a94ada92d2d119ffc19d3df26b1fc018ea5c391f148a92b02043: Status 404 returned error can't find the container with id 769887432b11a94ada92d2d119ffc19d3df26b1fc018ea5c391f148a92b02043
Nov 02 14:12:28.539042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:28.539012    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"]
Nov 02 14:12:29.058000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:29.057969    5034 generic.go:356] "Generic (PLEG): container finished" podID="8731eb60-f299-4c3c-8224-1b4a5f57a94c" containerID="7a804a2b2e0197615738518b496f195194310d446275fc099152adcd2b2ae8d2" exitCode=0
Nov 02 14:12:29.058446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:29.058065    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s" event={"ID":"8731eb60-f299-4c3c-8224-1b4a5f57a94c","Type":"ContainerDied","Data":"7a804a2b2e0197615738518b496f195194310d446275fc099152adcd2b2ae8d2"}
Nov 02 14:12:29.059538 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:29.059509    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v" event={"ID":"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56","Type":"ContainerStarted","Data":"769887432b11a94ada92d2d119ffc19d3df26b1fc018ea5c391f148a92b02043"}
Nov 02 14:12:30.077800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.077260    5034 generic.go:356] "Generic (PLEG): container finished" podID="2f0e18b7-26ee-46a7-aaf2-d16f46a25f56" containerID="841bfc74187476ef1b6ec74ecc562f282a50b58df525d037a40730b2754f607e" exitCode=0
Nov 02 14:12:30.077800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.077663    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v" event={"ID":"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56","Type":"ContainerDied","Data":"841bfc74187476ef1b6ec74ecc562f282a50b58df525d037a40730b2754f607e"}
Nov 02 14:12:30.280336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.279994    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:30.398830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.398746    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-xks-ca\") pod \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") "
Nov 02 14:12:30.398830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.398796    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-openshift-ca\") pod \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") "
Nov 02 14:12:30.398830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.398818    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8731eb60-f299-4c3c-8224-1b4a5f57a94c-kube-api-access-r7g78\" (UniqueName: \"kubernetes.io/projected/8731eb60-f299-4c3c-8224-1b4a5f57a94c-kube-api-access-r7g78\") pod \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\" (UID: \"8731eb60-f299-4c3c-8224-1b4a5f57a94c\") "
Nov 02 14:12:30.399197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.399165    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-openshift-ca" pod "8731eb60-f299-4c3c-8224-1b4a5f57a94c" (UID: "8731eb60-f299-4c3c-8224-1b4a5f57a94c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:30.399330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.399272    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:30.400981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.400953    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8731eb60-f299-4c3c-8224-1b4a5f57a94c-kube-api-access-r7g78" pod "8731eb60-f299-4c3c-8224-1b4a5f57a94c" (UID: "8731eb60-f299-4c3c-8224-1b4a5f57a94c"). InnerVolumeSpecName "kube-api-access-r7g78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:30.401059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.401034    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-xks-ca" pod "8731eb60-f299-4c3c-8224-1b4a5f57a94c" (UID: "8731eb60-f299-4c3c-8224-1b4a5f57a94c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:30.500896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.500865    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8731eb60-f299-4c3c-8224-1b4a5f57a94c-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:30.500896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:30.500898    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7g78\" (UniqueName: \"kubernetes.io/projected/8731eb60-f299-4c3c-8224-1b4a5f57a94c-kube-api-access-r7g78\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:31.087212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.087175    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"
Nov 02 14:12:31.087212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.087196    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s" event={"ID":"8731eb60-f299-4c3c-8224-1b4a5f57a94c","Type":"ContainerDied","Data":"2e86a5c75b4eef720ad50c56fcda9d354d8d74324721405689565e11ebc1dbd5"}
Nov 02 14:12:31.087759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.087252    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e86a5c75b4eef720ad50c56fcda9d354d8d74324721405689565e11ebc1dbd5"
Nov 02 14:12:31.263201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.263178    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:31.311112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.311084    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-kube-api-access-t6thv\" (UniqueName: \"kubernetes.io/projected/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-kube-api-access-t6thv\") pod \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") "
Nov 02 14:12:31.311291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.311158    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-xks-ca\") pod \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") "
Nov 02 14:12:31.311291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.311228    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-openshift-ca\") pod \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\" (UID: \"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56\") "
Nov 02 14:12:31.311577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.311551    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-openshift-ca" pod "2f0e18b7-26ee-46a7-aaf2-d16f46a25f56" (UID: "2f0e18b7-26ee-46a7-aaf2-d16f46a25f56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:31.311848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.311831    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:31.313252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.313227    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-xks-ca" pod "2f0e18b7-26ee-46a7-aaf2-d16f46a25f56" (UID: "2f0e18b7-26ee-46a7-aaf2-d16f46a25f56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:31.313374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.313359    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-kube-api-access-t6thv" pod "2f0e18b7-26ee-46a7-aaf2-d16f46a25f56" (UID: "2f0e18b7-26ee-46a7-aaf2-d16f46a25f56"). InnerVolumeSpecName "kube-api-access-t6thv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:31.413222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.413081    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6thv\" (UniqueName: \"kubernetes.io/projected/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-kube-api-access-t6thv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:31.413222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:31.413114    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:32.096598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:32.096564    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"
Nov 02 14:12:32.096598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:32.096581    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v" event={"ID":"2f0e18b7-26ee-46a7-aaf2-d16f46a25f56","Type":"ContainerDied","Data":"769887432b11a94ada92d2d119ffc19d3df26b1fc018ea5c391f148a92b02043"}
Nov 02 14:12:32.097093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:32.096642    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="769887432b11a94ada92d2d119ffc19d3df26b1fc018ea5c391f148a92b02043"
Nov 02 14:12:39.569416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.569382    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"]
Nov 02 14:12:39.570032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.570015    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8731eb60-f299-4c3c-8224-1b4a5f57a94c" containerName="revoke-keys"
Nov 02 14:12:39.570084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.570034    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8731eb60-f299-4c3c-8224-1b4a5f57a94c" containerName="revoke-keys"
Nov 02 14:12:39.570084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.570074    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2f0e18b7-26ee-46a7-aaf2-d16f46a25f56" containerName="revoke-keys"
Nov 02 14:12:39.570084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.570079    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f0e18b7-26ee-46a7-aaf2-d16f46a25f56" containerName="revoke-keys"
Nov 02 14:12:39.570186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.570180    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8731eb60-f299-4c3c-8224-1b4a5f57a94c" containerName="revoke-keys"
Nov 02 14:12:39.570221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.570194    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f0e18b7-26ee-46a7-aaf2-d16f46a25f56" containerName="revoke-keys"
Nov 02 14:12:39.575634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.575616    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.594694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.594669    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"]
Nov 02 14:12:39.602821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.602737    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:12:39.602821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.602725    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:12:39.603122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.603102    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:12:39.605856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.605812    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:12:39.653958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.653924    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af617526-7528d\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.654120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.654098    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af617526-7528d\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.654165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.654138    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkw4\" (UniqueName: \"kubernetes.io/projected/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-kube-api-access-2zkw4\") pod \"maas-api-revoke-sub-af617526-7528d\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.755419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.755394    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af617526-7528d\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.755549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.755507    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af617526-7528d\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.755549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.755545    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkw4\" (UniqueName: \"kubernetes.io/projected/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-kube-api-access-2zkw4\") pod \"maas-api-revoke-sub-af617526-7528d\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.756204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.756180    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af617526-7528d\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.757826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.757794    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af617526-7528d\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.780379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.780357    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkw4\" (UniqueName: \"kubernetes.io/projected/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-kube-api-access-2zkw4\") pod \"maas-api-revoke-sub-af617526-7528d\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:39.890143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:39.890058    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:40.219376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:40.219343    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"]
Nov 02 14:12:40.219857 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:12:40.219832    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2209c59_eb7c_4b99_adc1_a0133f6ca5b6.slice/crio-73dfd6bb59a83909767fd66855300a26544739e6711a41fe3877ba7e455d1ae8 WatchSource:0}: Error finding container 73dfd6bb59a83909767fd66855300a26544739e6711a41fe3877ba7e455d1ae8: Status 404 returned error can't find the container with id 73dfd6bb59a83909767fd66855300a26544739e6711a41fe3877ba7e455d1ae8
Nov 02 14:12:41.181024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:41.180985    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d" event={"ID":"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6","Type":"ContainerStarted","Data":"50def01b2cd6f022292b5df7c8b02739b8a7e48e9955cf912b996f6396509539"}
Nov 02 14:12:41.181411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:41.181051    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d" event={"ID":"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6","Type":"ContainerStarted","Data":"73dfd6bb59a83909767fd66855300a26544739e6711a41fe3877ba7e455d1ae8"}
Nov 02 14:12:41.227521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:41.227469    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d" podStartSLOduration=2.227455102 podStartE2EDuration="2.227455102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:41.225073303 +0000 UTC m=+1751.976377746" watchObservedRunningTime="2026-11-02 14:12:41.227455102 +0000 UTC m=+1751.978759540"
Nov 02 14:12:42.191588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:42.191556    5034 generic.go:356] "Generic (PLEG): container finished" podID="e2209c59-eb7c-4b99-adc1-a0133f6ca5b6" containerID="50def01b2cd6f022292b5df7c8b02739b8a7e48e9955cf912b996f6396509539" exitCode=0
Nov 02 14:12:42.192043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:42.191649    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d" event={"ID":"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6","Type":"ContainerDied","Data":"50def01b2cd6f022292b5df7c8b02739b8a7e48e9955cf912b996f6396509539"}
Nov 02 14:12:43.399958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.399927    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:43.508309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.508277    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-xks-ca\") pod \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") "
Nov 02 14:12:43.508490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.508418    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-openshift-ca\") pod \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") "
Nov 02 14:12:43.508554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.508514    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-kube-api-access-2zkw4\" (UniqueName: \"kubernetes.io/projected/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-kube-api-access-2zkw4\") pod \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\" (UID: \"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6\") "
Nov 02 14:12:43.508882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.508837    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-openshift-ca" pod "e2209c59-eb7c-4b99-adc1-a0133f6ca5b6" (UID: "e2209c59-eb7c-4b99-adc1-a0133f6ca5b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:43.509307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.509283    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:43.510932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.510901    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-xks-ca" pod "e2209c59-eb7c-4b99-adc1-a0133f6ca5b6" (UID: "e2209c59-eb7c-4b99-adc1-a0133f6ca5b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:43.511687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.511661    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-kube-api-access-2zkw4" pod "e2209c59-eb7c-4b99-adc1-a0133f6ca5b6" (UID: "e2209c59-eb7c-4b99-adc1-a0133f6ca5b6"). InnerVolumeSpecName "kube-api-access-2zkw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:43.611328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.611295    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zkw4\" (UniqueName: \"kubernetes.io/projected/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-kube-api-access-2zkw4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:43.611328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:43.611325    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:44.210614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.210572    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d" event={"ID":"e2209c59-eb7c-4b99-adc1-a0133f6ca5b6","Type":"ContainerDied","Data":"73dfd6bb59a83909767fd66855300a26544739e6711a41fe3877ba7e455d1ae8"}
Nov 02 14:12:44.210836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.210640    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73dfd6bb59a83909767fd66855300a26544739e6711a41fe3877ba7e455d1ae8"
Nov 02 14:12:44.210836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.210650    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"
Nov 02 14:12:44.391061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.391024    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"]
Nov 02 14:12:44.391712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.391691    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e2209c59-eb7c-4b99-adc1-a0133f6ca5b6" containerName="revoke-keys"
Nov 02 14:12:44.391712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.391713    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2209c59-eb7c-4b99-adc1-a0133f6ca5b6" containerName="revoke-keys"
Nov 02 14:12:44.391968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.391953    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2209c59-eb7c-4b99-adc1-a0133f6ca5b6" containerName="revoke-keys"
Nov 02 14:12:44.395691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.395671    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:44.413086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.413064    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:12:44.413584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.413086    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:12:44.413706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.413683    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:12:44.418697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.418669    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"]
Nov 02 14:12:44.426027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.426003    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:12:44.541709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.541318    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf725bc3-684qv\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:44.579005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.578944    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf725bc3-684qv\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:44.579388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.579365    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2zmq\" (UniqueName: \"kubernetes.io/projected/862142b3-9c3f-4315-8bc8-ed798b3cc675-kube-api-access-k2zmq\") pod \"maas-api-revoke-sub-cf725bc3-684qv\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:44.686800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.683667    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf725bc3-684qv\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:44.686800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.683733    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2zmq\" (UniqueName: \"kubernetes.io/projected/862142b3-9c3f-4315-8bc8-ed798b3cc675-kube-api-access-k2zmq\") pod \"maas-api-revoke-sub-cf725bc3-684qv\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:44.686800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.683848    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf725bc3-684qv\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:44.686800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.685727    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf725bc3-684qv\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:44.709495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.709462    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf725bc3-684qv\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:44.748241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:44.732117    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2zmq\" (UniqueName: \"kubernetes.io/projected/862142b3-9c3f-4315-8bc8-ed798b3cc675-kube-api-access-k2zmq\") pod \"maas-api-revoke-sub-cf725bc3-684qv\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:45.013298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:45.013255    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:45.456108 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:12:45.455975    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862142b3_9c3f_4315_8bc8_ed798b3cc675.slice/crio-a2dc8dfff189a57cb273f02e577928e11ea7a934772a53def017642bcff794c8 WatchSource:0}: Error finding container a2dc8dfff189a57cb273f02e577928e11ea7a934772a53def017642bcff794c8: Status 404 returned error can't find the container with id a2dc8dfff189a57cb273f02e577928e11ea7a934772a53def017642bcff794c8
Nov 02 14:12:45.464617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:45.464586    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"]
Nov 02 14:12:46.363801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:46.359253    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv" event={"ID":"862142b3-9c3f-4315-8bc8-ed798b3cc675","Type":"ContainerStarted","Data":"7c9881370d161fe85f9c186a3d384d77e79ce276d6b8778825a2c5b2f7248ec5"}
Nov 02 14:12:46.377758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:46.377724    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv" event={"ID":"862142b3-9c3f-4315-8bc8-ed798b3cc675","Type":"ContainerStarted","Data":"a2dc8dfff189a57cb273f02e577928e11ea7a934772a53def017642bcff794c8"}
Nov 02 14:12:46.477662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:46.477556    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv" podStartSLOduration=2.477538557 podStartE2EDuration="2.477538557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:46.445002667 +0000 UTC m=+1757.196307106" watchObservedRunningTime="2026-11-02 14:12:46.477538557 +0000 UTC m=+1757.228842997"
Nov 02 14:12:47.368168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.368132    5034 generic.go:356] "Generic (PLEG): container finished" podID="862142b3-9c3f-4315-8bc8-ed798b3cc675" containerID="7c9881370d161fe85f9c186a3d384d77e79ce276d6b8778825a2c5b2f7248ec5" exitCode=0
Nov 02 14:12:47.368381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.368206    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv" event={"ID":"862142b3-9c3f-4315-8bc8-ed798b3cc675","Type":"ContainerDied","Data":"7c9881370d161fe85f9c186a3d384d77e79ce276d6b8778825a2c5b2f7248ec5"}
Nov 02 14:12:47.638260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.638168    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"]
Nov 02 14:12:47.643331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.643302    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.663222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.663195    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"]
Nov 02 14:12:47.695319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.695280    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29349365-6f5hz\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.695606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.695588    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29349365-6f5hz\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.695718 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.695690    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2q5c\" (UniqueName: \"kubernetes.io/projected/4f91d65d-6e57-403b-91e3-cb641250aa69-kube-api-access-w2q5c\") pod \"maas-api-revoke-sub-29349365-6f5hz\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.797633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.797602    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2q5c\" (UniqueName: \"kubernetes.io/projected/4f91d65d-6e57-403b-91e3-cb641250aa69-kube-api-access-w2q5c\") pod \"maas-api-revoke-sub-29349365-6f5hz\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.797830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.797658    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29349365-6f5hz\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.797918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.797846    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29349365-6f5hz\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.798365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.798339    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29349365-6f5hz\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.800170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.800141    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29349365-6f5hz\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.845470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.845438    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2q5c\" (UniqueName: \"kubernetes.io/projected/4f91d65d-6e57-403b-91e3-cb641250aa69-kube-api-access-w2q5c\") pod \"maas-api-revoke-sub-29349365-6f5hz\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:47.958086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:47.958027    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:48.266169 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:12:48.266140    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f91d65d_6e57_403b_91e3_cb641250aa69.slice/crio-bf1b4af1ce740edebc6e700ecf9c1dcb0a7f8c9eb6eaa80e522f6cc2828cfe39 WatchSource:0}: Error finding container bf1b4af1ce740edebc6e700ecf9c1dcb0a7f8c9eb6eaa80e522f6cc2828cfe39: Status 404 returned error can't find the container with id bf1b4af1ce740edebc6e700ecf9c1dcb0a7f8c9eb6eaa80e522f6cc2828cfe39
Nov 02 14:12:48.268306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.268282    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"]
Nov 02 14:12:48.384605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.384571    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz" event={"ID":"4f91d65d-6e57-403b-91e3-cb641250aa69","Type":"ContainerStarted","Data":"bf1b4af1ce740edebc6e700ecf9c1dcb0a7f8c9eb6eaa80e522f6cc2828cfe39"}
Nov 02 14:12:48.413157 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:12:48.413118    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:12:48.553311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.553289    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:48.608404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.608375    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862142b3-9c3f-4315-8bc8-ed798b3cc675-kube-api-access-k2zmq\" (UniqueName: \"kubernetes.io/projected/862142b3-9c3f-4315-8bc8-ed798b3cc675-kube-api-access-k2zmq\") pod \"862142b3-9c3f-4315-8bc8-ed798b3cc675\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") "
Nov 02 14:12:48.608565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.608469    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-xks-ca\") pod \"862142b3-9c3f-4315-8bc8-ed798b3cc675\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") "
Nov 02 14:12:48.608565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.608504    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-openshift-ca\") pod \"862142b3-9c3f-4315-8bc8-ed798b3cc675\" (UID: \"862142b3-9c3f-4315-8bc8-ed798b3cc675\") "
Nov 02 14:12:48.608934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.608906    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-openshift-ca" pod "862142b3-9c3f-4315-8bc8-ed798b3cc675" (UID: "862142b3-9c3f-4315-8bc8-ed798b3cc675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:48.609091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.609076    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:48.669566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.669532    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-xks-ca" pod "862142b3-9c3f-4315-8bc8-ed798b3cc675" (UID: "862142b3-9c3f-4315-8bc8-ed798b3cc675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:48.669566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.669542    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862142b3-9c3f-4315-8bc8-ed798b3cc675-kube-api-access-k2zmq" pod "862142b3-9c3f-4315-8bc8-ed798b3cc675" (UID: "862142b3-9c3f-4315-8bc8-ed798b3cc675"). InnerVolumeSpecName "kube-api-access-k2zmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:48.710932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.710901    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2zmq\" (UniqueName: \"kubernetes.io/projected/862142b3-9c3f-4315-8bc8-ed798b3cc675-kube-api-access-k2zmq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:48.711060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:48.710936    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862142b3-9c3f-4315-8bc8-ed798b3cc675-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:49.396674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:49.396639    5034 generic.go:356] "Generic (PLEG): container finished" podID="4f91d65d-6e57-403b-91e3-cb641250aa69" containerID="7b1446d033186d2da92c92b86608103d9206a54a2932b1e0af6ea656db70a24a" exitCode=0
Nov 02 14:12:49.398480 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:49.398458    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"
Nov 02 14:12:49.399186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:49.399157    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz" event={"ID":"4f91d65d-6e57-403b-91e3-cb641250aa69","Type":"ContainerDied","Data":"7b1446d033186d2da92c92b86608103d9206a54a2932b1e0af6ea656db70a24a"}
Nov 02 14:12:49.399322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:49.399209    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv" event={"ID":"862142b3-9c3f-4315-8bc8-ed798b3cc675","Type":"ContainerDied","Data":"a2dc8dfff189a57cb273f02e577928e11ea7a934772a53def017642bcff794c8"}
Nov 02 14:12:49.399322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:49.399238    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2dc8dfff189a57cb273f02e577928e11ea7a934772a53def017642bcff794c8"
Nov 02 14:12:50.643459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.643423    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:50.745500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.745469    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f91d65d-6e57-403b-91e3-cb641250aa69-kube-api-access-w2q5c\" (UniqueName: \"kubernetes.io/projected/4f91d65d-6e57-403b-91e3-cb641250aa69-kube-api-access-w2q5c\") pod \"4f91d65d-6e57-403b-91e3-cb641250aa69\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") "
Nov 02 14:12:50.745676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.745568    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-xks-ca\") pod \"4f91d65d-6e57-403b-91e3-cb641250aa69\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") "
Nov 02 14:12:50.745731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.745713    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-openshift-ca\") pod \"4f91d65d-6e57-403b-91e3-cb641250aa69\" (UID: \"4f91d65d-6e57-403b-91e3-cb641250aa69\") "
Nov 02 14:12:50.746143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.746111    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-openshift-ca" pod "4f91d65d-6e57-403b-91e3-cb641250aa69" (UID: "4f91d65d-6e57-403b-91e3-cb641250aa69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:12:50.746291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.746260    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:50.747901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.747877    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f91d65d-6e57-403b-91e3-cb641250aa69-kube-api-access-w2q5c" pod "4f91d65d-6e57-403b-91e3-cb641250aa69" (UID: "4f91d65d-6e57-403b-91e3-cb641250aa69"). InnerVolumeSpecName "kube-api-access-w2q5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:12:50.748181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.748160    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-xks-ca" pod "4f91d65d-6e57-403b-91e3-cb641250aa69" (UID: "4f91d65d-6e57-403b-91e3-cb641250aa69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:12:50.847081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.847052    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2q5c\" (UniqueName: \"kubernetes.io/projected/4f91d65d-6e57-403b-91e3-cb641250aa69-kube-api-access-w2q5c\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:50.847081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:50.847078    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f91d65d-6e57-403b-91e3-cb641250aa69-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:12:51.418704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:51.418671    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"
Nov 02 14:12:51.418924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:51.418713    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz" event={"ID":"4f91d65d-6e57-403b-91e3-cb641250aa69","Type":"ContainerDied","Data":"bf1b4af1ce740edebc6e700ecf9c1dcb0a7f8c9eb6eaa80e522f6cc2828cfe39"}
Nov 02 14:12:51.418924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:51.418767    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf1b4af1ce740edebc6e700ecf9c1dcb0a7f8c9eb6eaa80e522f6cc2828cfe39"
Nov 02 14:12:58.357625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.357587    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"]
Nov 02 14:12:58.358457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.358440    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4f91d65d-6e57-403b-91e3-cb641250aa69" containerName="revoke-keys"
Nov 02 14:12:58.358535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.358458    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f91d65d-6e57-403b-91e3-cb641250aa69" containerName="revoke-keys"
Nov 02 14:12:58.358535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.358479    5034 cpu_manager.go:414] "containerMap: removing container" podUID="862142b3-9c3f-4315-8bc8-ed798b3cc675" containerName="revoke-keys"
Nov 02 14:12:58.358535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.358487    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862142b3-9c3f-4315-8bc8-ed798b3cc675" containerName="revoke-keys"
Nov 02 14:12:58.358681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.358669    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f91d65d-6e57-403b-91e3-cb641250aa69" containerName="revoke-keys"
Nov 02 14:12:58.358723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.358682    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="862142b3-9c3f-4315-8bc8-ed798b3cc675" containerName="revoke-keys"
Nov 02 14:12:58.362208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.362188    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.388289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.388250    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"]
Nov 02 14:12:58.392108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.392079    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:12:58.392247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.392192    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:12:58.423875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.423844    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:12:58.423875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.423860    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:12:58.514029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.513992    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af10ada4-zx9qn\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.514201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.514109    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrfw2\" (UniqueName: \"kubernetes.io/projected/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-kube-api-access-zrfw2\") pod \"maas-api-revoke-sub-af10ada4-zx9qn\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.514263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.514214    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af10ada4-zx9qn\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.616051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.615972    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af10ada4-zx9qn\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.616214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.616053    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrfw2\" (UniqueName: \"kubernetes.io/projected/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-kube-api-access-zrfw2\") pod \"maas-api-revoke-sub-af10ada4-zx9qn\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.616214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.616113    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af10ada4-zx9qn\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.616851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.616822    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af10ada4-zx9qn\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.618355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.618326    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af10ada4-zx9qn\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.652073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.652049    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrfw2\" (UniqueName: \"kubernetes.io/projected/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-kube-api-access-zrfw2\") pod \"maas-api-revoke-sub-af10ada4-zx9qn\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:58.677768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:58.677742    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:12:59.001309 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:12:59.001275    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76a7105_dc24_4bfc_ac67_09d4a1212ff7.slice/crio-de40b93436c23bced43185279c9cc3d8f3ee580b83af8654cac947fbdf113fd5 WatchSource:0}: Error finding container de40b93436c23bced43185279c9cc3d8f3ee580b83af8654cac947fbdf113fd5: Status 404 returned error can't find the container with id de40b93436c23bced43185279c9cc3d8f3ee580b83af8654cac947fbdf113fd5
Nov 02 14:12:59.004000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.003976    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"]
Nov 02 14:12:59.432858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.432828    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"]
Nov 02 14:12:59.438231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.438214    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:12:59.454986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.454967    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-cdaf64-dockercfg-l8rnh\""
Nov 02 14:12:59.459713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.459669    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"]
Nov 02 14:12:59.468719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.468168    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-cdaf64\""
Nov 02 14:12:59.484864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.484836    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn" event={"ID":"b76a7105-dc24-4bfc-ac67-09d4a1212ff7","Type":"ContainerStarted","Data":"6c57bfc996708e01bfb1c2841f2dd53ce462344a5ba533e82ac7c127fc952760"}
Nov 02 14:12:59.485142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.485032    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn" event={"ID":"b76a7105-dc24-4bfc-ac67-09d4a1212ff7","Type":"ContainerStarted","Data":"de40b93436c23bced43185279c9cc3d8f3ee580b83af8654cac947fbdf113fd5"}
Nov 02 14:12:59.566801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.557883    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cprx\" (UniqueName: \"kubernetes.io/projected/e7f497c9-0630-457f-99e9-5e125a849bfa-kube-api-access-9cprx\") pod \"payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd\" (UID: \"e7f497c9-0630-457f-99e9-5e125a849bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:12:59.566801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.558147    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f497c9-0630-457f-99e9-5e125a849bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd\" (UID: \"e7f497c9-0630-457f-99e9-5e125a849bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:12:59.611853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.609892    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"]
Nov 02 14:12:59.625006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.623411    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn" podStartSLOduration=1.623389359 podStartE2EDuration="1.623389359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:12:59.59015734 +0000 UTC m=+1770.341461783" watchObservedRunningTime="2026-11-02 14:12:59.623389359 +0000 UTC m=+1770.374693801"
Nov 02 14:12:59.625006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.624498    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"]
Nov 02 14:12:59.625006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.624703    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:12:59.660731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.659865    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f497c9-0630-457f-99e9-5e125a849bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd\" (UID: \"e7f497c9-0630-457f-99e9-5e125a849bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:12:59.660731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.660046    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cprx\" (UniqueName: \"kubernetes.io/projected/e7f497c9-0630-457f-99e9-5e125a849bfa-kube-api-access-9cprx\") pod \"payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd\" (UID: \"e7f497c9-0630-457f-99e9-5e125a849bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:12:59.661039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.661014    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f497c9-0630-457f-99e9-5e125a849bfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd\" (UID: \"e7f497c9-0630-457f-99e9-5e125a849bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:12:59.684724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.684698    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cprx\" (UniqueName: \"kubernetes.io/projected/e7f497c9-0630-457f-99e9-5e125a849bfa-kube-api-access-9cprx\") pod \"payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd\" (UID: \"e7f497c9-0630-457f-99e9-5e125a849bfa\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:12:59.755555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.755528    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:12:59.773118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.761429    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7836e47-b5cc-4583-9869-8555040ecd1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct\" (UID: \"d7836e47-b5cc-4583-9869-8555040ecd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:12:59.773118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.761647    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqh9v\" (UniqueName: \"kubernetes.io/projected/d7836e47-b5cc-4583-9869-8555040ecd1f-kube-api-access-dqh9v\") pod \"payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct\" (UID: \"d7836e47-b5cc-4583-9869-8555040ecd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:12:59.862644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.862611    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqh9v\" (UniqueName: \"kubernetes.io/projected/d7836e47-b5cc-4583-9869-8555040ecd1f-kube-api-access-dqh9v\") pod \"payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct\" (UID: \"d7836e47-b5cc-4583-9869-8555040ecd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:12:59.866969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.865556    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7836e47-b5cc-4583-9869-8555040ecd1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct\" (UID: \"d7836e47-b5cc-4583-9869-8555040ecd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:12:59.866969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.866913    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7836e47-b5cc-4583-9869-8555040ecd1f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct\" (UID: \"d7836e47-b5cc-4583-9869-8555040ecd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:12:59.895647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.895513    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqh9v\" (UniqueName: \"kubernetes.io/projected/d7836e47-b5cc-4583-9869-8555040ecd1f-kube-api-access-dqh9v\") pod \"payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct\" (UID: \"d7836e47-b5cc-4583-9869-8555040ecd1f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:12:59.915330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.915045    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 14:12:59.915330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.915102    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 14:12:59.952132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:12:59.952104    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:13:00.094293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.094254    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"]
Nov 02 14:13:00.126271 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:00.126240    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f497c9_0630_457f_99e9_5e125a849bfa.slice/crio-3d3bb1239cd939b2c183d7982c651760ac57bb5ffcf6524637711bb9adc8f520 WatchSource:0}: Error finding container 3d3bb1239cd939b2c183d7982c651760ac57bb5ffcf6524637711bb9adc8f520: Status 404 returned error can't find the container with id 3d3bb1239cd939b2c183d7982c651760ac57bb5ffcf6524637711bb9adc8f520
Nov 02 14:13:00.258212 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:00.258178    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7836e47_b5cc_4583_9869_8555040ecd1f.slice/crio-ddac59c6a8e8e53d6ebe98417e218361e80e3a27589e8de8dcc3de0ea6ee1dc2 WatchSource:0}: Error finding container ddac59c6a8e8e53d6ebe98417e218361e80e3a27589e8de8dcc3de0ea6ee1dc2: Status 404 returned error can't find the container with id ddac59c6a8e8e53d6ebe98417e218361e80e3a27589e8de8dcc3de0ea6ee1dc2
Nov 02 14:13:00.259551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.259526    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"]
Nov 02 14:13:00.494009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.493978    5034 generic.go:356] "Generic (PLEG): container finished" podID="b69662c7-cd64-4242-84fb-bd6deea012be" containerID="c59ae6cb7f477809dfeb513f2aeaf5ac4b80849419bae81064b12d394e1c6c40" exitCode=0
Nov 02 14:13:00.494444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.494052    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" event={"ID":"b69662c7-cd64-4242-84fb-bd6deea012be","Type":"ContainerDied","Data":"c59ae6cb7f477809dfeb513f2aeaf5ac4b80849419bae81064b12d394e1c6c40"}
Nov 02 14:13:00.494501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.494486    5034 scope.go:122] "RemoveContainer" containerID="c59ae6cb7f477809dfeb513f2aeaf5ac4b80849419bae81064b12d394e1c6c40"
Nov 02 14:13:00.495933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.495903    5034 generic.go:356] "Generic (PLEG): container finished" podID="b76a7105-dc24-4bfc-ac67-09d4a1212ff7" containerID="6c57bfc996708e01bfb1c2841f2dd53ce462344a5ba533e82ac7c127fc952760" exitCode=0
Nov 02 14:13:00.496068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.495942    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn" event={"ID":"b76a7105-dc24-4bfc-ac67-09d4a1212ff7","Type":"ContainerDied","Data":"6c57bfc996708e01bfb1c2841f2dd53ce462344a5ba533e82ac7c127fc952760"}
Nov 02 14:13:00.496164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.496138    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:00.496232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.496221    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:00.496294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.496258    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:00.498284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.498253    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct" event={"ID":"d7836e47-b5cc-4583-9869-8555040ecd1f","Type":"ContainerStarted","Data":"e9c7a9ccfa6959ff958c9bc783545233272c68d25f0bc5f5e2803df0b5b31ecf"}
Nov 02 14:13:00.498390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.498306    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct" event={"ID":"d7836e47-b5cc-4583-9869-8555040ecd1f","Type":"ContainerStarted","Data":"ddac59c6a8e8e53d6ebe98417e218361e80e3a27589e8de8dcc3de0ea6ee1dc2"}
Nov 02 14:13:00.498461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.498409    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:13:00.500312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.500289    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd" event={"ID":"e7f497c9-0630-457f-99e9-5e125a849bfa","Type":"ContainerStarted","Data":"ce15ebe2c23b02ac3117e1a95e8e87549d02854b52682e5f0a73fb4503ba11e5"}
Nov 02 14:13:00.500415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.500329    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd" event={"ID":"e7f497c9-0630-457f-99e9-5e125a849bfa","Type":"ContainerStarted","Data":"3d3bb1239cd939b2c183d7982c651760ac57bb5ffcf6524637711bb9adc8f520"}
Nov 02 14:13:00.500415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.500348    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:13:00.596169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.591871    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd" podStartSLOduration=1.591853186 podStartE2EDuration="1.591853186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:00.587391269 +0000 UTC m=+1771.338695708" watchObservedRunningTime="2026-11-02 14:13:00.591853186 +0000 UTC m=+1771.343157622"
Nov 02 14:13:00.680180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:00.680121    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct" podStartSLOduration=1.6801027830000002 podStartE2EDuration="1.680102783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:00.678971756 +0000 UTC m=+1771.430276189" watchObservedRunningTime="2026-11-02 14:13:00.680102783 +0000 UTC m=+1771.431407222"
Nov 02 14:13:01.512392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:01.512356    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" event={"ID":"b69662c7-cd64-4242-84fb-bd6deea012be","Type":"ContainerStarted","Data":"0f4c1650dd17b719470a10e82b6c8f86495a53db2b0352a2480472a30caecdff"}
Nov 02 14:13:01.815114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:01.810276    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:13:01.915521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:01.915365    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-xks-ca\") pod \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") "
Nov 02 14:13:01.915735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:01.915530    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-kube-api-access-zrfw2\" (UniqueName: \"kubernetes.io/projected/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-kube-api-access-zrfw2\") pod \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") "
Nov 02 14:13:01.915735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:01.915679    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-openshift-ca\") pod \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\" (UID: \"b76a7105-dc24-4bfc-ac67-09d4a1212ff7\") "
Nov 02 14:13:01.916135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:01.916107    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-openshift-ca" pod "b76a7105-dc24-4bfc-ac67-09d4a1212ff7" (UID: "b76a7105-dc24-4bfc-ac67-09d4a1212ff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:01.916254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:01.916235    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:01.919364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:01.919333    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-xks-ca" pod "b76a7105-dc24-4bfc-ac67-09d4a1212ff7" (UID: "b76a7105-dc24-4bfc-ac67-09d4a1212ff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:01.919943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:01.919917    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-kube-api-access-zrfw2" pod "b76a7105-dc24-4bfc-ac67-09d4a1212ff7" (UID: "b76a7105-dc24-4bfc-ac67-09d4a1212ff7"). InnerVolumeSpecName "kube-api-access-zrfw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:02.017808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:02.017762    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:02.017808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:02.017806    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrfw2\" (UniqueName: \"kubernetes.io/projected/b76a7105-dc24-4bfc-ac67-09d4a1212ff7-kube-api-access-zrfw2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:02.212161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:02.212070    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:13:02.212376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:02.212350    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 14:13:02.212440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:02.212413    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 14:13:02.524111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:02.524082    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"
Nov 02 14:13:02.524522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:02.524110    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn" event={"ID":"b76a7105-dc24-4bfc-ac67-09d4a1212ff7","Type":"ContainerDied","Data":"de40b93436c23bced43185279c9cc3d8f3ee580b83af8654cac947fbdf113fd5"}
Nov 02 14:13:02.524522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:02.524169    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de40b93436c23bced43185279c9cc3d8f3ee580b83af8654cac947fbdf113fd5"
Nov 02 14:13:03.212344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:03.212309    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 14:13:03.212529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:03.212367    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 14:13:04.212358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.212319    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 14:13:04.212809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.212403    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 14:13:04.655424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.655392    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"]
Nov 02 14:13:04.656054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.656040    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b76a7105-dc24-4bfc-ac67-09d4a1212ff7" containerName="revoke-keys"
Nov 02 14:13:04.656111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.656055    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76a7105-dc24-4bfc-ac67-09d4a1212ff7" containerName="revoke-keys"
Nov 02 14:13:04.656233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.656223    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76a7105-dc24-4bfc-ac67-09d4a1212ff7" containerName="revoke-keys"
Nov 02 14:13:04.660041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.660024    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:04.708806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.707085    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:13:04.708806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.707138    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:13:04.708806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.707771    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:13:04.710921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.710894    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"]
Nov 02 14:13:04.756398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.756285    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5524fc-cs782\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:04.756526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.756420    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5524fc-cs782\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:04.756526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.756459    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgxx\" (UniqueName: \"kubernetes.io/projected/28827380-d430-4957-a4ad-ab59e6c4646f-kube-api-access-qzgxx\") pod \"maas-api-revoke-sub-4c5524fc-cs782\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:04.793145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.792506    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:13:04.857622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.857586    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5524fc-cs782\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:04.857842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.857658    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgxx\" (UniqueName: \"kubernetes.io/projected/28827380-d430-4957-a4ad-ab59e6c4646f-kube-api-access-qzgxx\") pod \"maas-api-revoke-sub-4c5524fc-cs782\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:04.857947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.857929    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5524fc-cs782\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:04.858469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.858435    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c5524fc-cs782\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:04.861146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.861116    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c5524fc-cs782\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:04.891799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:04.889456    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgxx\" (UniqueName: \"kubernetes.io/projected/28827380-d430-4957-a4ad-ab59e6c4646f-kube-api-access-qzgxx\") pod \"maas-api-revoke-sub-4c5524fc-cs782\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:05.011661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.011630    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:05.122711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.122675    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"]
Nov 02 14:13:05.134310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.129476    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.148324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.148292    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"]
Nov 02 14:13:05.161844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.161817    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f70ff89b-d6mhx\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.162073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.162050    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f70ff89b-d6mhx\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.162142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.162087    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5lm\" (UniqueName: \"kubernetes.io/projected/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-kube-api-access-vz5lm\") pod \"maas-api-revoke-sub-f70ff89b-d6mhx\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.212810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.212443    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 14:13:05.212810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.212496    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 14:13:05.263318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.263230    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f70ff89b-d6mhx\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.263485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.263427    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f70ff89b-d6mhx\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.263485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.263454    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5lm\" (UniqueName: \"kubernetes.io/projected/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-kube-api-access-vz5lm\") pod \"maas-api-revoke-sub-f70ff89b-d6mhx\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.264085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.264065    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f70ff89b-d6mhx\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.265583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.265566    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f70ff89b-d6mhx\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.306473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.306444    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5lm\" (UniqueName: \"kubernetes.io/projected/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-kube-api-access-vz5lm\") pod \"maas-api-revoke-sub-f70ff89b-d6mhx\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.348952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.348824    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"]
Nov 02 14:13:05.491460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.491432    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:05.545417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.545346    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782" event={"ID":"28827380-d430-4957-a4ad-ab59e6c4646f","Type":"ContainerStarted","Data":"1c606bfa2261bb5fab105949da57903b839888364dcde5caf25f21dda687c891"}
Nov 02 14:13:05.910821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:05.909547    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"]
Nov 02 14:13:05.911582 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:05.911556    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd80e0281_ec3a_4ad1_a29f_262ccf7a408e.slice/crio-4a45295c9fb0c9835e6167f7c5f7bf1d0ae348746f661f0d3a3f9bb252bcf987 WatchSource:0}: Error finding container 4a45295c9fb0c9835e6167f7c5f7bf1d0ae348746f661f0d3a3f9bb252bcf987: Status 404 returned error can't find the container with id 4a45295c9fb0c9835e6167f7c5f7bf1d0ae348746f661f0d3a3f9bb252bcf987
Nov 02 14:13:06.213889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:06.213844    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Nov 02 14:13:06.215634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:06.215443    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Nov 02 14:13:06.554372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:06.554334    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782" event={"ID":"28827380-d430-4957-a4ad-ab59e6c4646f","Type":"ContainerStarted","Data":"3756e4da9c6d5dbae2eafb8608c38cad2757208aa174d09141b76bb0fd5b74cc"}
Nov 02 14:13:06.556092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:06.556067    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx" event={"ID":"d80e0281-ec3a-4ad1-a29f-262ccf7a408e","Type":"ContainerStarted","Data":"8e5469121ea12d48d4c6b73e63c12e9dddf1a51ccf7d6f60953c9ca0276bc061"}
Nov 02 14:13:06.556226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:06.556110    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx" event={"ID":"d80e0281-ec3a-4ad1-a29f-262ccf7a408e","Type":"ContainerStarted","Data":"4a45295c9fb0c9835e6167f7c5f7bf1d0ae348746f661f0d3a3f9bb252bcf987"}
Nov 02 14:13:06.607506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:06.607444    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782" podStartSLOduration=2.60742509 podStartE2EDuration="2.60742509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:06.599186625 +0000 UTC m=+1777.350491063" watchObservedRunningTime="2026-11-02 14:13:06.60742509 +0000 UTC m=+1777.358729530"
Nov 02 14:13:06.648497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:06.648404    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx" podStartSLOduration=1.648385487 podStartE2EDuration="1.648385487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:06.642910773 +0000 UTC m=+1777.394215210" watchObservedRunningTime="2026-11-02 14:13:06.648385487 +0000 UTC m=+1777.399689931"
Nov 02 14:13:06.805266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:06.805225    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"]
Nov 02 14:13:06.826431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:06.826401    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"]
Nov 02 14:13:07.135926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.135892    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"]
Nov 02 14:13:07.139768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.139743    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.160275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.160248    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"]
Nov 02 14:13:07.217489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.217455    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:13:07.218849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.218824    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:13:07.219555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.219533    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:13:07.236387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.236361    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8x8l\" (UniqueName: \"kubernetes.io/projected/f55d1057-cc5f-4e2f-8c1e-943223596e1e-kube-api-access-f8x8l\") pod \"maas-api-revoke-sub-48377d11-mlllh\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.236488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.236428    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48377d11-mlllh\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.236564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.236548    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48377d11-mlllh\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.340799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.338060    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8x8l\" (UniqueName: \"kubernetes.io/projected/f55d1057-cc5f-4e2f-8c1e-943223596e1e-kube-api-access-f8x8l\") pod \"maas-api-revoke-sub-48377d11-mlllh\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.340799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.338165    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48377d11-mlllh\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.340799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.338289    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48377d11-mlllh\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.340799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.339866    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48377d11-mlllh\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.359606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.359576    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48377d11-mlllh\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.382962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.382932    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8x8l\" (UniqueName: \"kubernetes.io/projected/f55d1057-cc5f-4e2f-8c1e-943223596e1e-kube-api-access-f8x8l\") pod \"maas-api-revoke-sub-48377d11-mlllh\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.459123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.459079    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:07.577882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.575411    5034 generic.go:356] "Generic (PLEG): container finished" podID="d80e0281-ec3a-4ad1-a29f-262ccf7a408e" containerID="8e5469121ea12d48d4c6b73e63c12e9dddf1a51ccf7d6f60953c9ca0276bc061" exitCode=0
Nov 02 14:13:07.577882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.575571    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx" event={"ID":"d80e0281-ec3a-4ad1-a29f-262ccf7a408e","Type":"ContainerDied","Data":"8e5469121ea12d48d4c6b73e63c12e9dddf1a51ccf7d6f60953c9ca0276bc061"}
Nov 02 14:13:07.578164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.578121    5034 generic.go:356] "Generic (PLEG): container finished" podID="28827380-d430-4957-a4ad-ab59e6c4646f" containerID="3756e4da9c6d5dbae2eafb8608c38cad2757208aa174d09141b76bb0fd5b74cc" exitCode=0
Nov 02 14:13:07.579153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.579121    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782" event={"ID":"28827380-d430-4957-a4ad-ab59e6c4646f","Type":"ContainerDied","Data":"3756e4da9c6d5dbae2eafb8608c38cad2757208aa174d09141b76bb0fd5b74cc"}
Nov 02 14:13:07.881769 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:07.881741    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55d1057_cc5f_4e2f_8c1e_943223596e1e.slice/crio-35f5b7a1a5d08e588cd74ed7f5feaf79e23aed822604ae32ec783c3ad6169954 WatchSource:0}: Error finding container 35f5b7a1a5d08e588cd74ed7f5feaf79e23aed822604ae32ec783c3ad6169954: Status 404 returned error can't find the container with id 35f5b7a1a5d08e588cd74ed7f5feaf79e23aed822604ae32ec783c3ad6169954
Nov 02 14:13:07.883415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:07.883378    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"]
Nov 02 14:13:08.592673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:08.592559    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh" event={"ID":"f55d1057-cc5f-4e2f-8c1e-943223596e1e","Type":"ContainerStarted","Data":"e13fd2ba43c9ab330b5ff9c774b4c398665cc1888b20b88e879b832fd80c6837"}
Nov 02 14:13:08.594050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:08.593351    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh" event={"ID":"f55d1057-cc5f-4e2f-8c1e-943223596e1e","Type":"ContainerStarted","Data":"35f5b7a1a5d08e588cd74ed7f5feaf79e23aed822604ae32ec783c3ad6169954"}
Nov 02 14:13:08.645722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:08.641899    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh" podStartSLOduration=1.641879828 podStartE2EDuration="1.641879828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:08.637195576 +0000 UTC m=+1779.388500017" watchObservedRunningTime="2026-11-02 14:13:08.641879828 +0000 UTC m=+1779.393184269"
Nov 02 14:13:09.257224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.256866    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:09.435131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.432278    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-kube-api-access-vz5lm\" (UniqueName: \"kubernetes.io/projected/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-kube-api-access-vz5lm\") pod \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") "
Nov 02 14:13:09.435131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.432373    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-openshift-ca\") pod \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") "
Nov 02 14:13:09.435131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.432451    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-xks-ca\") pod \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\" (UID: \"d80e0281-ec3a-4ad1-a29f-262ccf7a408e\") "
Nov 02 14:13:09.446985 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.446396    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-openshift-ca" pod "d80e0281-ec3a-4ad1-a29f-262ccf7a408e" (UID: "d80e0281-ec3a-4ad1-a29f-262ccf7a408e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:09.639853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.622042    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-xks-ca" pod "d80e0281-ec3a-4ad1-a29f-262ccf7a408e" (UID: "d80e0281-ec3a-4ad1-a29f-262ccf7a408e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:09.659798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.656281    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:09.659798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.656323    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:09.713860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.707228    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-kube-api-access-vz5lm" pod "d80e0281-ec3a-4ad1-a29f-262ccf7a408e" (UID: "d80e0281-ec3a-4ad1-a29f-262ccf7a408e"). InnerVolumeSpecName "kube-api-access-vz5lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:09.729212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.729127    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:09.730000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.729927    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782" event={"ID":"28827380-d430-4957-a4ad-ab59e6c4646f","Type":"ContainerDied","Data":"1c606bfa2261bb5fab105949da57903b839888364dcde5caf25f21dda687c891"}
Nov 02 14:13:09.730000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.729981    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c606bfa2261bb5fab105949da57903b839888364dcde5caf25f21dda687c891"
Nov 02 14:13:09.732261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.732239    5034 generic.go:356] "Generic (PLEG): container finished" podID="f55d1057-cc5f-4e2f-8c1e-943223596e1e" containerID="e13fd2ba43c9ab330b5ff9c774b4c398665cc1888b20b88e879b832fd80c6837" exitCode=0
Nov 02 14:13:09.732500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.732484    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh" event={"ID":"f55d1057-cc5f-4e2f-8c1e-943223596e1e","Type":"ContainerDied","Data":"e13fd2ba43c9ab330b5ff9c774b4c398665cc1888b20b88e879b832fd80c6837"}
Nov 02 14:13:09.734218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.734199    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx" event={"ID":"d80e0281-ec3a-4ad1-a29f-262ccf7a408e","Type":"ContainerDied","Data":"4a45295c9fb0c9835e6167f7c5f7bf1d0ae348746f661f0d3a3f9bb252bcf987"}
Nov 02 14:13:09.734393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.734357    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a45295c9fb0c9835e6167f7c5f7bf1d0ae348746f661f0d3a3f9bb252bcf987"
Nov 02 14:13:09.734527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.734516    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"
Nov 02 14:13:09.763756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.763550    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5lm\" (UniqueName: \"kubernetes.io/projected/d80e0281-ec3a-4ad1-a29f-262ccf7a408e-kube-api-access-vz5lm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:09.895857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.876918    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-openshift-ca\") pod \"28827380-d430-4957-a4ad-ab59e6c4646f\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") "
Nov 02 14:13:09.896200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.896169    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-xks-ca\") pod \"28827380-d430-4957-a4ad-ab59e6c4646f\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") "
Nov 02 14:13:09.896345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.896329    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28827380-d430-4957-a4ad-ab59e6c4646f-kube-api-access-qzgxx\" (UniqueName: \"kubernetes.io/projected/28827380-d430-4957-a4ad-ab59e6c4646f-kube-api-access-qzgxx\") pod \"28827380-d430-4957-a4ad-ab59e6c4646f\" (UID: \"28827380-d430-4957-a4ad-ab59e6c4646f\") "
Nov 02 14:13:09.911542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.911510    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-openshift-ca" pod "28827380-d430-4957-a4ad-ab59e6c4646f" (UID: "28827380-d430-4957-a4ad-ab59e6c4646f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:09.914655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.914613    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-xks-ca" pod "28827380-d430-4957-a4ad-ab59e6c4646f" (UID: "28827380-d430-4957-a4ad-ab59e6c4646f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:09.918031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:09.917993    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28827380-d430-4957-a4ad-ab59e6c4646f-kube-api-access-qzgxx" pod "28827380-d430-4957-a4ad-ab59e6c4646f" (UID: "28827380-d430-4957-a4ad-ab59e6c4646f"). InnerVolumeSpecName "kube-api-access-qzgxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:10.010882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:10.010810    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:10.010882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:10.010841    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28827380-d430-4957-a4ad-ab59e6c4646f-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:10.010882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:10.010856    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzgxx\" (UniqueName: \"kubernetes.io/projected/28827380-d430-4957-a4ad-ab59e6c4646f-kube-api-access-qzgxx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:10.741939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:10.741902    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"
Nov 02 14:13:10.917341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:10.917316    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:11.027327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.027240    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55d1057-cc5f-4e2f-8c1e-943223596e1e-kube-api-access-f8x8l\" (UniqueName: \"kubernetes.io/projected/f55d1057-cc5f-4e2f-8c1e-943223596e1e-kube-api-access-f8x8l\") pod \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") "
Nov 02 14:13:11.027497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.027452    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-openshift-ca\") pod \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") "
Nov 02 14:13:11.027571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.027553    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-xks-ca\") pod \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\" (UID: \"f55d1057-cc5f-4e2f-8c1e-943223596e1e\") "
Nov 02 14:13:11.027741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.027716    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-openshift-ca" pod "f55d1057-cc5f-4e2f-8c1e-943223596e1e" (UID: "f55d1057-cc5f-4e2f-8c1e-943223596e1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:11.028021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.028001    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:11.029360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.029333    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55d1057-cc5f-4e2f-8c1e-943223596e1e-kube-api-access-f8x8l" pod "f55d1057-cc5f-4e2f-8c1e-943223596e1e" (UID: "f55d1057-cc5f-4e2f-8c1e-943223596e1e"). InnerVolumeSpecName "kube-api-access-f8x8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:11.029543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.029522    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-xks-ca" pod "f55d1057-cc5f-4e2f-8c1e-943223596e1e" (UID: "f55d1057-cc5f-4e2f-8c1e-943223596e1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:11.129085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.129053    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8x8l\" (UniqueName: \"kubernetes.io/projected/f55d1057-cc5f-4e2f-8c1e-943223596e1e-kube-api-access-f8x8l\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:11.129085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.129081    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f55d1057-cc5f-4e2f-8c1e-943223596e1e-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:11.516196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.516171    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:13:11.516388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.516286    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:13:11.750152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.750127    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"
Nov 02 14:13:11.750152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.750136    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh" event={"ID":"f55d1057-cc5f-4e2f-8c1e-943223596e1e","Type":"ContainerDied","Data":"35f5b7a1a5d08e588cd74ed7f5feaf79e23aed822604ae32ec783c3ad6169954"}
Nov 02 14:13:11.750587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.750190    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f5b7a1a5d08e588cd74ed7f5feaf79e23aed822604ae32ec783c3ad6169954"
Nov 02 14:13:11.805663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.805629    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm" podUID="4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" containerName="payload-processing" containerID="cri-o://ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e" gracePeriod=25
Nov 02 14:13:11.826748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:11.826700    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q" podUID="b96268ea-20ee-4a56-abe0-1d20f56dcecd" containerName="payload-pre-processing" containerID="cri-o://d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1" gracePeriod=25
Nov 02 14:13:16.495300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.495267    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"]
Nov 02 14:13:16.496239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.496217    5034 cpu_manager.go:414] "containerMap: removing container" podUID="28827380-d430-4957-a4ad-ab59e6c4646f" containerName="revoke-keys"
Nov 02 14:13:16.496239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.496240    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28827380-d430-4957-a4ad-ab59e6c4646f" containerName="revoke-keys"
Nov 02 14:13:16.496380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.496294    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d80e0281-ec3a-4ad1-a29f-262ccf7a408e" containerName="revoke-keys"
Nov 02 14:13:16.496380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.496303    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d80e0281-ec3a-4ad1-a29f-262ccf7a408e" containerName="revoke-keys"
Nov 02 14:13:16.496380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.496330    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f55d1057-cc5f-4e2f-8c1e-943223596e1e" containerName="revoke-keys"
Nov 02 14:13:16.496380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.496335    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55d1057-cc5f-4e2f-8c1e-943223596e1e" containerName="revoke-keys"
Nov 02 14:13:16.496527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.496445    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="28827380-d430-4957-a4ad-ab59e6c4646f" containerName="revoke-keys"
Nov 02 14:13:16.496527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.496465    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d80e0281-ec3a-4ad1-a29f-262ccf7a408e" containerName="revoke-keys"
Nov 02 14:13:16.496527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.496481    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55d1057-cc5f-4e2f-8c1e-943223596e1e" containerName="revoke-keys"
Nov 02 14:13:16.501410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.501390    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.532390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.532361    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"]
Nov 02 14:13:16.606214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.606182    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.606370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.606226    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfqvx\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-kube-api-access-vfqvx\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.606370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.606343    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.606444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.606395    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.606507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.606485    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.606581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.606559    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.606796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.606599    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.606796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.606616    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.606907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.606788    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.708479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708454    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.708635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708494    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.708635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708519    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.708635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708543    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.708635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708573    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.708635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708589    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.708916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708703    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.708916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708834    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.708916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708882    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfqvx\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-kube-api-access-vfqvx\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.709116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.708943    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.709932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.709889    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.710204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.710174    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.710368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.710314    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.710859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.710832    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.711759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.711726    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.716032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.716007    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.733400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.733375    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.733947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.733928    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfqvx\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-kube-api-access-vfqvx\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:16.816155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:16.816090    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:17.151580 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:17.151542    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1ff927_cad5_4b00_b3ae_39d9de8ce94c.slice/crio-a9292e72d948b30353ad947ee7ad23ba86dca335742244282e585faf866fc7f9 WatchSource:0}: Error finding container a9292e72d948b30353ad947ee7ad23ba86dca335742244282e585faf866fc7f9: Status 404 returned error can't find the container with id a9292e72d948b30353ad947ee7ad23ba86dca335742244282e585faf866fc7f9
Nov 02 14:13:17.154049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:17.154023    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"]
Nov 02 14:13:17.154662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:17.154630    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:17.154723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:17.154694    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:17.154758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:17.154735    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:17.808617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:17.808576    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" event={"ID":"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c","Type":"ContainerStarted","Data":"68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c"}
Nov 02 14:13:17.809137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:17.808642    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" event={"ID":"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c","Type":"ContainerStarted","Data":"a9292e72d948b30353ad947ee7ad23ba86dca335742244282e585faf866fc7f9"}
Nov 02 14:13:17.816592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:17.816571    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:17.895189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:17.894139    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" podStartSLOduration=1.894114306 podStartE2EDuration="1.894114306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:17.86870218 +0000 UTC m=+1788.620006631" watchObservedRunningTime="2026-11-02 14:13:17.894114306 +0000 UTC m=+1788.645418744"
Nov 02 14:13:18.423055 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:13:18.423010    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:13:18.818585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:18.818555    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 14:13:18.819006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:18.818606    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 14:13:19.790174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:19.789893    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q" podUID="b96268ea-20ee-4a56-abe0-1d20f56dcecd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:13:19.790435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:19.790411    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm" podUID="4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:13:19.817089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:19.817060    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 14:13:19.817242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:19.817116    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 14:13:20.817334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:20.816913    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 14:13:20.817334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:20.816976    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 14:13:21.702983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.702942    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"]
Nov 02 14:13:21.708765 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.708738    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:21.709508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.709472    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx99q\" (UniqueName: \"kubernetes.io/projected/de80a299-3630-4777-a161-69b39ecae74a-kube-api-access-fx99q\") pod \"maas-api-revoke-sub-175e7bfd-j29m9\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:21.709638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.709605    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de80a299-3630-4777-a161-69b39ecae74a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175e7bfd-j29m9\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:21.709706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.709691    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de80a299-3630-4777-a161-69b39ecae74a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175e7bfd-j29m9\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:21.725282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.725258    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"]
Nov 02 14:13:21.733206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.733176    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:13:21.733415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.733398    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:13:21.773073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.773038    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:13:21.819494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.773408    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:13:21.820049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.812940    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx99q\" (UniqueName: \"kubernetes.io/projected/de80a299-3630-4777-a161-69b39ecae74a-kube-api-access-fx99q\") pod \"maas-api-revoke-sub-175e7bfd-j29m9\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:21.820274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.820257    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de80a299-3630-4777-a161-69b39ecae74a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175e7bfd-j29m9\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:21.820448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.820434    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de80a299-3630-4777-a161-69b39ecae74a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175e7bfd-j29m9\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:21.836968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.823221    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de80a299-3630-4777-a161-69b39ecae74a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175e7bfd-j29m9\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:21.836968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.825287    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 14:13:21.836968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.825330    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 14:13:21.840931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.840897    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de80a299-3630-4777-a161-69b39ecae74a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175e7bfd-j29m9\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:21.848506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:21.848450    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx99q\" (UniqueName: \"kubernetes.io/projected/de80a299-3630-4777-a161-69b39ecae74a-kube-api-access-fx99q\") pod \"maas-api-revoke-sub-175e7bfd-j29m9\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:22.030437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:22.030404    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:22.402153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:22.401008    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"]
Nov 02 14:13:22.405189 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:22.405164    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde80a299_3630_4777_a161_69b39ecae74a.slice/crio-ef638b4e5f8e3b1b0aae3f1589b9baded7dbefe1f5aaa53bba69b5011e7f454e WatchSource:0}: Error finding container ef638b4e5f8e3b1b0aae3f1589b9baded7dbefe1f5aaa53bba69b5011e7f454e: Status 404 returned error can't find the container with id ef638b4e5f8e3b1b0aae3f1589b9baded7dbefe1f5aaa53bba69b5011e7f454e
Nov 02 14:13:22.817224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:22.817189    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 14:13:22.817411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:22.817382    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 14:13:22.864726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:22.864702    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9" event={"ID":"de80a299-3630-4777-a161-69b39ecae74a","Type":"ContainerStarted","Data":"3edbbe8920abadca619a701b9bb7ed07cca2441305528bce8a691bcf66da027e"}
Nov 02 14:13:22.865087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:22.864745    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9" event={"ID":"de80a299-3630-4777-a161-69b39ecae74a","Type":"ContainerStarted","Data":"ef638b4e5f8e3b1b0aae3f1589b9baded7dbefe1f5aaa53bba69b5011e7f454e"}
Nov 02 14:13:22.926203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:22.926144    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9" podStartSLOduration=1.926126472 podStartE2EDuration="1.926126472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:22.92307686 +0000 UTC m=+1793.674381299" watchObservedRunningTime="2026-11-02 14:13:22.926126472 +0000 UTC m=+1793.677430911"
Nov 02 14:13:23.816808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:23.816712    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 02 14:13:23.816983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:23.816773    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 02 14:13:23.877479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:23.877446    5034 generic.go:356] "Generic (PLEG): container finished" podID="de80a299-3630-4777-a161-69b39ecae74a" containerID="3edbbe8920abadca619a701b9bb7ed07cca2441305528bce8a691bcf66da027e" exitCode=0
Nov 02 14:13:23.877968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:23.877532    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9" event={"ID":"de80a299-3630-4777-a161-69b39ecae74a","Type":"ContainerDied","Data":"3edbbe8920abadca619a701b9bb7ed07cca2441305528bce8a691bcf66da027e"}
Nov 02 14:13:24.821838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:24.821796    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:24.822145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:24.822127    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:24.822931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:24.822910    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:13:25.169613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.169588    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:25.282655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.282619    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de80a299-3630-4777-a161-69b39ecae74a-kube-api-access-fx99q\" (UniqueName: \"kubernetes.io/projected/de80a299-3630-4777-a161-69b39ecae74a-kube-api-access-fx99q\") pod \"de80a299-3630-4777-a161-69b39ecae74a\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") "
Nov 02 14:13:25.282889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.282691    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de80a299-3630-4777-a161-69b39ecae74a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de80a299-3630-4777-a161-69b39ecae74a-maas-api-xks-ca\") pod \"de80a299-3630-4777-a161-69b39ecae74a\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") "
Nov 02 14:13:25.282889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.282821    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de80a299-3630-4777-a161-69b39ecae74a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de80a299-3630-4777-a161-69b39ecae74a-maas-api-openshift-ca\") pod \"de80a299-3630-4777-a161-69b39ecae74a\" (UID: \"de80a299-3630-4777-a161-69b39ecae74a\") "
Nov 02 14:13:25.283255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.283224    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de80a299-3630-4777-a161-69b39ecae74a-maas-api-openshift-ca" pod "de80a299-3630-4777-a161-69b39ecae74a" (UID: "de80a299-3630-4777-a161-69b39ecae74a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:25.283440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.283423    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de80a299-3630-4777-a161-69b39ecae74a-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:25.285581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.285498    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de80a299-3630-4777-a161-69b39ecae74a-kube-api-access-fx99q" pod "de80a299-3630-4777-a161-69b39ecae74a" (UID: "de80a299-3630-4777-a161-69b39ecae74a"). InnerVolumeSpecName "kube-api-access-fx99q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:25.285693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.285568    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de80a299-3630-4777-a161-69b39ecae74a-maas-api-xks-ca" pod "de80a299-3630-4777-a161-69b39ecae74a" (UID: "de80a299-3630-4777-a161-69b39ecae74a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:25.385194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.385165    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx99q\" (UniqueName: \"kubernetes.io/projected/de80a299-3630-4777-a161-69b39ecae74a-kube-api-access-fx99q\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:25.385194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.385192    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de80a299-3630-4777-a161-69b39ecae74a-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:25.894695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.894667    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"
Nov 02 14:13:25.894927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.894726    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9" event={"ID":"de80a299-3630-4777-a161-69b39ecae74a","Type":"ContainerDied","Data":"ef638b4e5f8e3b1b0aae3f1589b9baded7dbefe1f5aaa53bba69b5011e7f454e"}
Nov 02 14:13:25.894927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:25.894765    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef638b4e5f8e3b1b0aae3f1589b9baded7dbefe1f5aaa53bba69b5011e7f454e"
Nov 02 14:13:28.915891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.915854    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"]
Nov 02 14:13:28.916879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.916856    5034 cpu_manager.go:414] "containerMap: removing container" podUID="de80a299-3630-4777-a161-69b39ecae74a" containerName="revoke-keys"
Nov 02 14:13:28.916981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.916881    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de80a299-3630-4777-a161-69b39ecae74a" containerName="revoke-keys"
Nov 02 14:13:28.917175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.917156    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="de80a299-3630-4777-a161-69b39ecae74a" containerName="revoke-keys"
Nov 02 14:13:28.921724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.921701    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:28.944408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.944381    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"]
Nov 02 14:13:28.952898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.951264    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:13:28.952898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.951317    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:13:28.952898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.951430    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:13:28.952898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:28.951611    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:13:29.059521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.059454    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fad5039-nsbgh\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.059731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.059564    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb9pm\" (UniqueName: \"kubernetes.io/projected/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-kube-api-access-kb9pm\") pod \"maas-api-revoke-sub-6fad5039-nsbgh\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.059731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.059637    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fad5039-nsbgh\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.161885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.161840    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fad5039-nsbgh\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.163513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.163476    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fad5039-nsbgh\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.163762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.163731    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb9pm\" (UniqueName: \"kubernetes.io/projected/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-kube-api-access-kb9pm\") pod \"maas-api-revoke-sub-6fad5039-nsbgh\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.164318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.163116    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fad5039-nsbgh\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.201838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.192276    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fad5039-nsbgh\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.202049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.195428    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb9pm\" (UniqueName: \"kubernetes.io/projected/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-kube-api-access-kb9pm\") pod \"maas-api-revoke-sub-6fad5039-nsbgh\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.246989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.246966    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:29.857992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.857538    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm" podUID="4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:13:29.858268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:29.858244    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q" podUID="b96268ea-20ee-4a56-abe0-1d20f56dcecd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:13:30.178803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.178722    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"]
Nov 02 14:13:30.250850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.250791    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:13:30.255672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.255650    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:13:30.715919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.715851    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"]
Nov 02 14:13:30.720148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.720127    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:30.741302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.741273    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"]
Nov 02 14:13:30.896248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.896214    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abb03689-zljcv\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:30.896418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.896392    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abb03689-zljcv\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:30.896465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.896438    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xkfm\" (UniqueName: \"kubernetes.io/projected/30321d02-bf32-4cbe-aebb-54e11d957a8f-kube-api-access-6xkfm\") pod \"maas-api-revoke-sub-abb03689-zljcv\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:30.997962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.997923    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abb03689-zljcv\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:30.998137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.998000    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xkfm\" (UniqueName: \"kubernetes.io/projected/30321d02-bf32-4cbe-aebb-54e11d957a8f-kube-api-access-6xkfm\") pod \"maas-api-revoke-sub-abb03689-zljcv\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:30.998214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.998200    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abb03689-zljcv\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:30.998820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:30.998770    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-abb03689-zljcv\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:31.000308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:31.000283    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-abb03689-zljcv\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:31.022657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:31.022632    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xkfm\" (UniqueName: \"kubernetes.io/projected/30321d02-bf32-4cbe-aebb-54e11d957a8f-kube-api-access-6xkfm\") pod \"maas-api-revoke-sub-abb03689-zljcv\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:31.034552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:31.034524    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:31.203938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:31.203906    5034 generic.go:356] "Generic (PLEG): container finished" podID="fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e" containerID="d257e02e4eee521852630f1c68e083e51519235d6236df45d605027d614a3545" exitCode=0
Nov 02 14:13:31.204345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:31.203991    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh" event={"ID":"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e","Type":"ContainerDied","Data":"d257e02e4eee521852630f1c68e083e51519235d6236df45d605027d614a3545"}
Nov 02 14:13:31.204345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:31.204058    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh" event={"ID":"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e","Type":"ContainerStarted","Data":"6a57ce14b1c2bffdf658f605f818603253ff156e0a1d2dbc04abeff20629ab49"}
Nov 02 14:13:31.343952 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:31.343917    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30321d02_bf32_4cbe_aebb_54e11d957a8f.slice/crio-8984669d76036c75e81984e3208917ca0c8b2c970e5486dea82f7a71c85ce3be WatchSource:0}: Error finding container 8984669d76036c75e81984e3208917ca0c8b2c970e5486dea82f7a71c85ce3be: Status 404 returned error can't find the container with id 8984669d76036c75e81984e3208917ca0c8b2c970e5486dea82f7a71c85ce3be
Nov 02 14:13:31.345943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:31.345915    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"]
Nov 02 14:13:32.160319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.160292    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:13:32.171954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.171930    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:13:32.228791 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.228741    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-kube-api-access-hcj28\" (UniqueName: \"kubernetes.io/projected/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-kube-api-access-hcj28\") pod \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\" (UID: \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\") "
Nov 02 14:13:32.229223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.228856    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-plugins-config-volume\") pod \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\" (UID: \"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec\") "
Nov 02 14:13:32.229223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.228930    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96268ea-20ee-4a56-abe0-1d20f56dcecd-kube-api-access-tbmx9\" (UniqueName: \"kubernetes.io/projected/b96268ea-20ee-4a56-abe0-1d20f56dcecd-kube-api-access-tbmx9\") pod \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\" (UID: \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\") "
Nov 02 14:13:32.229223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.229086    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96268ea-20ee-4a56-abe0-1d20f56dcecd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96268ea-20ee-4a56-abe0-1d20f56dcecd-plugins-config-volume\") pod \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\" (UID: \"b96268ea-20ee-4a56-abe0-1d20f56dcecd\") "
Nov 02 14:13:32.241666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.241627    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-plugins-config-volume" pod "4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" (UID: "4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:32.249876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.249839    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96268ea-20ee-4a56-abe0-1d20f56dcecd-kube-api-access-tbmx9" pod "b96268ea-20ee-4a56-abe0-1d20f56dcecd" (UID: "b96268ea-20ee-4a56-abe0-1d20f56dcecd"). InnerVolumeSpecName "kube-api-access-tbmx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:32.251133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.251101    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96268ea-20ee-4a56-abe0-1d20f56dcecd-plugins-config-volume" pod "b96268ea-20ee-4a56-abe0-1d20f56dcecd" (UID: "b96268ea-20ee-4a56-abe0-1d20f56dcecd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:32.254345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.254315    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-kube-api-access-hcj28" pod "4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" (UID: "4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec"). InnerVolumeSpecName "kube-api-access-hcj28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:32.300119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.292250    5034 generic.go:356] "Generic (PLEG): container finished" podID="b96268ea-20ee-4a56-abe0-1d20f56dcecd" containerID="d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1" exitCode=0
Nov 02 14:13:32.300119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.292515    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q" event={"ID":"b96268ea-20ee-4a56-abe0-1d20f56dcecd","Type":"ContainerDied","Data":"d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1"}
Nov 02 14:13:32.300119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.292621    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q" event={"ID":"b96268ea-20ee-4a56-abe0-1d20f56dcecd","Type":"ContainerDied","Data":"2f555c37580cfc08402f7f23d87289a6bf5942485f1f5601065cddaada9201ec"}
Nov 02 14:13:32.300119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.292667    5034 scope.go:122] "RemoveContainer" containerID="d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1"
Nov 02 14:13:32.300119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.296927    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"
Nov 02 14:13:32.373317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.371136    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.373317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.371173    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmx9\" (UniqueName: \"kubernetes.io/projected/b96268ea-20ee-4a56-abe0-1d20f56dcecd-kube-api-access-tbmx9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.373317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.371188    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b96268ea-20ee-4a56-abe0-1d20f56dcecd-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.373317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.371209    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcj28\" (UniqueName: \"kubernetes.io/projected/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec-kube-api-access-hcj28\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.376395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.376359    5034 generic.go:356] "Generic (PLEG): container finished" podID="4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" containerID="ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e" exitCode=0
Nov 02 14:13:32.377034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.376607    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"
Nov 02 14:13:32.378335 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.378071    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm" event={"ID":"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec","Type":"ContainerDied","Data":"ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e"}
Nov 02 14:13:32.378335 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.378130    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm" event={"ID":"4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec","Type":"ContainerDied","Data":"3282525313d3a0e5c2ab3fb9718fdd83c0103713bfa89d885578ff3de2abd65f"}
Nov 02 14:13:32.380161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.379353    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv" event={"ID":"30321d02-bf32-4cbe-aebb-54e11d957a8f","Type":"ContainerStarted","Data":"307343b76c1ec17c953e8e4c3d6701fb915aa1e65e50e5525e0e988a48d0fd99"}
Nov 02 14:13:32.380161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.379401    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv" event={"ID":"30321d02-bf32-4cbe-aebb-54e11d957a8f","Type":"ContainerStarted","Data":"8984669d76036c75e81984e3208917ca0c8b2c970e5486dea82f7a71c85ce3be"}
Nov 02 14:13:32.394029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.393993    5034 scope.go:122] "RemoveContainer" containerID="d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1"
Nov 02 14:13:32.395048 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:13:32.394661    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1\": container with ID starting with d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1 not found: ID does not exist" containerID="d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1"
Nov 02 14:13:32.395048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.394706    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1"} err="failed to get container status \"d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1\": rpc error: code = NotFound desc = could not find container \"d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1\": container with ID starting with d44153ac3ede252d55ffd7ebe7bacd2a70da1aa98efccc88d284eecfb8f46cb1 not found: ID does not exist"
Nov 02 14:13:32.395048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.394731    5034 scope.go:122] "RemoveContainer" containerID="ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e"
Nov 02 14:13:32.419465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.419438    5034 scope.go:122] "RemoveContainer" containerID="ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e"
Nov 02 14:13:32.419878 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:13:32.419839    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e\": container with ID starting with ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e not found: ID does not exist" containerID="ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e"
Nov 02 14:13:32.420002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.419889    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e"} err="failed to get container status \"ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e\": rpc error: code = NotFound desc = could not find container \"ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e\": container with ID starting with ef0b057df7b9a6e1d31901ee2008d598ef8b8dfcbc68837432f15f636cac5d8e not found: ID does not exist"
Nov 02 14:13:32.450469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.450417    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv" podStartSLOduration=2.450400937 podStartE2EDuration="2.450400937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:32.447287941 +0000 UTC m=+1803.198592380" watchObservedRunningTime="2026-11-02 14:13:32.450400937 +0000 UTC m=+1803.201705375"
Nov 02 14:13:32.493287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.493185    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"]
Nov 02 14:13:32.514094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.514060    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8acd34-6f94c7674f-t79dm"]
Nov 02 14:13:32.551980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.551951    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"]
Nov 02 14:13:32.555182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.555161    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:32.558691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.558668    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8acd34-66885b6cf-lnx9q"]
Nov 02 14:13:32.574711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.574683    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-openshift-ca\") pod \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") "
Nov 02 14:13:32.574856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.574743    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-xks-ca\") pod \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") "
Nov 02 14:13:32.574935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.574878    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-kube-api-access-kb9pm\" (UniqueName: \"kubernetes.io/projected/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-kube-api-access-kb9pm\") pod \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\" (UID: \"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e\") "
Nov 02 14:13:32.575119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.575092    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-openshift-ca" pod "fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e" (UID: "fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:32.575486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.575467    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.576815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.576769    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-kube-api-access-kb9pm" pod "fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e" (UID: "fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e"). InnerVolumeSpecName "kube-api-access-kb9pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:32.576899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.576885    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-xks-ca" pod "fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e" (UID: "fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:32.676338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.676255    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:32.676338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:32.676282    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb9pm\" (UniqueName: \"kubernetes.io/projected/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e-kube-api-access-kb9pm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:33.392313 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:33.392171    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"
Nov 02 14:13:33.394074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:33.394041    5034 generic.go:356] "Generic (PLEG): container finished" podID="30321d02-bf32-4cbe-aebb-54e11d957a8f" containerID="307343b76c1ec17c953e8e4c3d6701fb915aa1e65e50e5525e0e988a48d0fd99" exitCode=0
Nov 02 14:13:33.398473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:33.398439    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" path="/var/lib/kubelet/pods/4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec/volumes"
Nov 02 14:13:33.399091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:33.399069    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96268ea-20ee-4a56-abe0-1d20f56dcecd" path="/var/lib/kubelet/pods/b96268ea-20ee-4a56-abe0-1d20f56dcecd/volumes"
Nov 02 14:13:33.399754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:33.399721    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh" event={"ID":"fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e","Type":"ContainerDied","Data":"6a57ce14b1c2bffdf658f605f818603253ff156e0a1d2dbc04abeff20629ab49"}
Nov 02 14:13:33.399856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:33.399767    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a57ce14b1c2bffdf658f605f818603253ff156e0a1d2dbc04abeff20629ab49"
Nov 02 14:13:33.399856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:33.399798    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv" event={"ID":"30321d02-bf32-4cbe-aebb-54e11d957a8f","Type":"ContainerDied","Data":"307343b76c1ec17c953e8e4c3d6701fb915aa1e65e50e5525e0e988a48d0fd99"}
Nov 02 14:13:34.187648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.187618    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"]
Nov 02 14:13:34.188244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.188230    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" containerName="payload-processing"
Nov 02 14:13:34.188294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.188246    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" containerName="payload-processing"
Nov 02 14:13:34.188294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.188263    5034 cpu_manager.go:414] "containerMap: removing container" podUID="fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e" containerName="revoke-keys"
Nov 02 14:13:34.188294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.188269    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e" containerName="revoke-keys"
Nov 02 14:13:34.188294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.188288    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b96268ea-20ee-4a56-abe0-1d20f56dcecd" containerName="payload-pre-processing"
Nov 02 14:13:34.188294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.188293    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96268ea-20ee-4a56-abe0-1d20f56dcecd" containerName="payload-pre-processing"
Nov 02 14:13:34.188466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.188419    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba2ee0b-ba3f-47d8-9a73-2c99c2bec7ec" containerName="payload-processing"
Nov 02 14:13:34.188466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.188430    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e" containerName="revoke-keys"
Nov 02 14:13:34.188466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.188440    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96268ea-20ee-4a56-abe0-1d20f56dcecd" containerName="payload-pre-processing"
Nov 02 14:13:34.194001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.193985    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.212363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.212291    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"]
Nov 02 14:13:34.296851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.296819    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nzmb\" (UniqueName: \"kubernetes.io/projected/733f02af-e007-4937-b4ef-02fa3c04ce0f-kube-api-access-5nzmb\") pod \"maas-api-revoke-sub-e96abda6-4pnnm\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.297026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.296869    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e96abda6-4pnnm\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.297026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.297016    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e96abda6-4pnnm\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.398903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.398872    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e96abda6-4pnnm\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.399514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.399486    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nzmb\" (UniqueName: \"kubernetes.io/projected/733f02af-e007-4937-b4ef-02fa3c04ce0f-kube-api-access-5nzmb\") pod \"maas-api-revoke-sub-e96abda6-4pnnm\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.399848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.399824    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e96abda6-4pnnm\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.400610 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.400581    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e96abda6-4pnnm\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.402646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.402624    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e96abda6-4pnnm\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.424362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.424336    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nzmb\" (UniqueName: \"kubernetes.io/projected/733f02af-e007-4937-b4ef-02fa3c04ce0f-kube-api-access-5nzmb\") pod \"maas-api-revoke-sub-e96abda6-4pnnm\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.508548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.508505    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:34.600033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.600004    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:34.703171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.703135    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30321d02-bf32-4cbe-aebb-54e11d957a8f-kube-api-access-6xkfm\" (UniqueName: \"kubernetes.io/projected/30321d02-bf32-4cbe-aebb-54e11d957a8f-kube-api-access-6xkfm\") pod \"30321d02-bf32-4cbe-aebb-54e11d957a8f\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") "
Nov 02 14:13:34.703359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.703345    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-xks-ca\") pod \"30321d02-bf32-4cbe-aebb-54e11d957a8f\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") "
Nov 02 14:13:34.703435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.703383    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-openshift-ca\") pod \"30321d02-bf32-4cbe-aebb-54e11d957a8f\" (UID: \"30321d02-bf32-4cbe-aebb-54e11d957a8f\") "
Nov 02 14:13:34.703883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.703852    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-openshift-ca" pod "30321d02-bf32-4cbe-aebb-54e11d957a8f" (UID: "30321d02-bf32-4cbe-aebb-54e11d957a8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:34.704036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.704016    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:34.705337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.705297    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30321d02-bf32-4cbe-aebb-54e11d957a8f-kube-api-access-6xkfm" pod "30321d02-bf32-4cbe-aebb-54e11d957a8f" (UID: "30321d02-bf32-4cbe-aebb-54e11d957a8f"). InnerVolumeSpecName "kube-api-access-6xkfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:34.705460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.705352    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-xks-ca" pod "30321d02-bf32-4cbe-aebb-54e11d957a8f" (UID: "30321d02-bf32-4cbe-aebb-54e11d957a8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:34.805249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.805218    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xkfm\" (UniqueName: \"kubernetes.io/projected/30321d02-bf32-4cbe-aebb-54e11d957a8f-kube-api-access-6xkfm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:34.805249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.805249    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30321d02-bf32-4cbe-aebb-54e11d957a8f-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:34.842525 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:34.842492    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733f02af_e007_4937_b4ef_02fa3c04ce0f.slice/crio-9ffd705953ac3787ad808cda260052b574702c51cd21821013b10642f880d8be WatchSource:0}: Error finding container 9ffd705953ac3787ad808cda260052b574702c51cd21821013b10642f880d8be: Status 404 returned error can't find the container with id 9ffd705953ac3787ad808cda260052b574702c51cd21821013b10642f880d8be
Nov 02 14:13:34.844051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:34.844028    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"]
Nov 02 14:13:35.413071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:35.412970    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm" event={"ID":"733f02af-e007-4937-b4ef-02fa3c04ce0f","Type":"ContainerStarted","Data":"7f7296b27507ddb1268b50aeab436e67d5a3e48918a456d781d45889739c9192"}
Nov 02 14:13:35.413071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:35.413033    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm" event={"ID":"733f02af-e007-4937-b4ef-02fa3c04ce0f","Type":"ContainerStarted","Data":"9ffd705953ac3787ad808cda260052b574702c51cd21821013b10642f880d8be"}
Nov 02 14:13:35.414574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:35.414548    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv" event={"ID":"30321d02-bf32-4cbe-aebb-54e11d957a8f","Type":"ContainerDied","Data":"8984669d76036c75e81984e3208917ca0c8b2c970e5486dea82f7a71c85ce3be"}
Nov 02 14:13:35.414663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:35.414588    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8984669d76036c75e81984e3208917ca0c8b2c970e5486dea82f7a71c85ce3be"
Nov 02 14:13:35.414663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:35.414592    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"
Nov 02 14:13:35.458511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:35.458463    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm" podStartSLOduration=1.4584504200000001 podStartE2EDuration="1.45845042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:35.455721996 +0000 UTC m=+1806.207026435" watchObservedRunningTime="2026-11-02 14:13:35.45845042 +0000 UTC m=+1806.209754858"
Nov 02 14:13:36.423120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:36.423090    5034 generic.go:356] "Generic (PLEG): container finished" podID="733f02af-e007-4937-b4ef-02fa3c04ce0f" containerID="7f7296b27507ddb1268b50aeab436e67d5a3e48918a456d781d45889739c9192" exitCode=0
Nov 02 14:13:36.423538 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:36.423178    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm" event={"ID":"733f02af-e007-4937-b4ef-02fa3c04ce0f","Type":"ContainerDied","Data":"7f7296b27507ddb1268b50aeab436e67d5a3e48918a456d781d45889739c9192"}
Nov 02 14:13:37.590992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.590968    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:37.646416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.641885    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-openshift-ca\") pod \"733f02af-e007-4937-b4ef-02fa3c04ce0f\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") "
Nov 02 14:13:37.646416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.641983    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733f02af-e007-4937-b4ef-02fa3c04ce0f-kube-api-access-5nzmb\" (UniqueName: \"kubernetes.io/projected/733f02af-e007-4937-b4ef-02fa3c04ce0f-kube-api-access-5nzmb\") pod \"733f02af-e007-4937-b4ef-02fa3c04ce0f\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") "
Nov 02 14:13:37.646416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.642169    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-xks-ca\") pod \"733f02af-e007-4937-b4ef-02fa3c04ce0f\" (UID: \"733f02af-e007-4937-b4ef-02fa3c04ce0f\") "
Nov 02 14:13:37.646416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.642288    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-openshift-ca" pod "733f02af-e007-4937-b4ef-02fa3c04ce0f" (UID: "733f02af-e007-4937-b4ef-02fa3c04ce0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:37.646416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.642852    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:37.656385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.656356    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733f02af-e007-4937-b4ef-02fa3c04ce0f-kube-api-access-5nzmb" pod "733f02af-e007-4937-b4ef-02fa3c04ce0f" (UID: "733f02af-e007-4937-b4ef-02fa3c04ce0f"). InnerVolumeSpecName "kube-api-access-5nzmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:37.656526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.656497    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-xks-ca" pod "733f02af-e007-4937-b4ef-02fa3c04ce0f" (UID: "733f02af-e007-4937-b4ef-02fa3c04ce0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:37.743914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.743889    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nzmb\" (UniqueName: \"kubernetes.io/projected/733f02af-e007-4937-b4ef-02fa3c04ce0f-kube-api-access-5nzmb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:37.743914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:37.743913    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/733f02af-e007-4937-b4ef-02fa3c04ce0f-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:38.439031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.439004    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"
Nov 02 14:13:38.439197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.439043    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm" event={"ID":"733f02af-e007-4937-b4ef-02fa3c04ce0f","Type":"ContainerDied","Data":"9ffd705953ac3787ad808cda260052b574702c51cd21821013b10642f880d8be"}
Nov 02 14:13:38.439197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.439097    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ffd705953ac3787ad808cda260052b574702c51cd21821013b10642f880d8be"
Nov 02 14:13:38.490022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.489988    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"]
Nov 02 14:13:38.490218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.490190    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy" containerID="cri-o://ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2" gracePeriod=30
Nov 02 14:13:38.568211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.568176    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"]
Nov 02 14:13:38.568826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.568804    5034 cpu_manager.go:414] "containerMap: removing container" podUID="30321d02-bf32-4cbe-aebb-54e11d957a8f" containerName="revoke-keys"
Nov 02 14:13:38.568826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.568826    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30321d02-bf32-4cbe-aebb-54e11d957a8f" containerName="revoke-keys"
Nov 02 14:13:38.568962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.568848    5034 cpu_manager.go:414] "containerMap: removing container" podUID="733f02af-e007-4937-b4ef-02fa3c04ce0f" containerName="revoke-keys"
Nov 02 14:13:38.568962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.568855    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733f02af-e007-4937-b4ef-02fa3c04ce0f" containerName="revoke-keys"
Nov 02 14:13:38.569054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.568986    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="733f02af-e007-4937-b4ef-02fa3c04ce0f" containerName="revoke-keys"
Nov 02 14:13:38.569054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.569004    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="30321d02-bf32-4cbe-aebb-54e11d957a8f" containerName="revoke-keys"
Nov 02 14:13:38.574317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.574283    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.615551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.615515    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-cdaf64-openshift-default-dockercfg-g8696\""
Nov 02 14:13:38.700252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.700162    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4czm\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-kube-api-access-m4czm\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.700572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.700548    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.700729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.700710    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.700895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.700879    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d47968-77aa-4e90-8b68-d057588f5bef-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.701041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.701012    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d47968-77aa-4e90-8b68-d057588f5bef-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.701343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.701256    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.701519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.701503    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.701623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.701611    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.701880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.701862    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.706288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.706266    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"]
Nov 02 14:13:38.711645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.711620    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.715890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.715859    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"]
Nov 02 14:13:38.725260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.725231    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"]
Nov 02 14:13:38.802903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.802874    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d47968-77aa-4e90-8b68-d057588f5bef-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.803103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.802923    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d47968-77aa-4e90-8b68-d057588f5bef-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.803103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803013    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.803103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803052    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.803285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803101    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.803285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803131    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.803285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803157    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.803285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803227    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.803490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803287    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.803490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803318    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.803490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803381    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.803490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803429    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.803490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803473    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.803752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803527    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4czm\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-kube-api-access-m4czm\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.803752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803566    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.803752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803590    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v748\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-kube-api-access-2v748\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.803752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803638    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.803752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803670    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.804060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.803762    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d47968-77aa-4e90-8b68-d057588f5bef-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.804060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.804020    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.804060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.804037    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.804497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.804464    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.804627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.804512    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.806425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.806399    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d47968-77aa-4e90-8b68-d057588f5bef-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.806559 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.806458    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.829626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.829597    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.829831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.829812    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4czm\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-kube-api-access-m4czm\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:38.905320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905289    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.905455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905338    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.905455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905388    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v748\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-kube-api-access-2v748\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.905455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905431    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.905637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905546    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.905637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905609    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.905752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905678    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.905752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905712    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-data\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.905752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905748    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.905940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.905797    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.906094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.906072    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-certs\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.906246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.906224    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-credential-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.906462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.906345    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-socket\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.906593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.906505    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istiod-ca-cert\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.908605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.908573    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-envoy\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.908721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.908666    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-podinfo\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.930623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.930599    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-token\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:38.930856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:38.930840    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v748\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-kube-api-access-2v748\") pod \"e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") " pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:39.006669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.006643    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:39.044115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.044091    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:39.366128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.366100    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"]
Nov 02 14:13:39.368237 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:39.368215    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d47968_77aa_4e90_8b68_d057588f5bef.slice/crio-37bd3a4ac77939579ce1e5fb8e68de7e71e5ca0914a7d8142c6f20111143ea9f WatchSource:0}: Error finding container 37bd3a4ac77939579ce1e5fb8e68de7e71e5ca0914a7d8142c6f20111143ea9f: Status 404 returned error can't find the container with id 37bd3a4ac77939579ce1e5fb8e68de7e71e5ca0914a7d8142c6f20111143ea9f
Nov 02 14:13:39.371195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.371157    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:39.371298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.371241    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:39.371298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.371282    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:39.394078 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:39.393892    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5486ccc_2e7b_427a_bc0b_8c6245dca2f4.slice/crio-d6b9c06f787ec0aaa374318cc9b1a337e26393cb17fa37b39982b39de843854c WatchSource:0}: Error finding container d6b9c06f787ec0aaa374318cc9b1a337e26393cb17fa37b39982b39de843854c: Status 404 returned error can't find the container with id d6b9c06f787ec0aaa374318cc9b1a337e26393cb17fa37b39982b39de843854c
Nov 02 14:13:39.396283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.396231    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:39.396383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.396317    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:39.396383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.396360    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:39.399286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.399262    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"]
Nov 02 14:13:39.448694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.448664    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" event={"ID":"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4","Type":"ContainerStarted","Data":"d6b9c06f787ec0aaa374318cc9b1a337e26393cb17fa37b39982b39de843854c"}
Nov 02 14:13:39.450653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:39.450627    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" event={"ID":"61d47968-77aa-4e90-8b68-d057588f5bef","Type":"ContainerStarted","Data":"37bd3a4ac77939579ce1e5fb8e68de7e71e5ca0914a7d8142c6f20111143ea9f"}
Nov 02 14:13:40.460491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:40.460452    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" event={"ID":"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4","Type":"ContainerStarted","Data":"2f1a16efd621cf3ddc459928ecaffa01ca4802b2b47c7e5c44ff6643b77c698c"}
Nov 02 14:13:40.462310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:40.462284    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" event={"ID":"61d47968-77aa-4e90-8b68-d057588f5bef","Type":"ContainerStarted","Data":"33549632a99f551df2578c2f84cc6aef7a495b50f0090a6af93e336910c4a25d"}
Nov 02 14:13:40.542633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:40.542523    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podStartSLOduration=2.5425039959999998 podStartE2EDuration="2.542503996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:40.537073653 +0000 UTC m=+1811.288378103" watchObservedRunningTime="2026-11-02 14:13:40.542503996 +0000 UTC m=+1811.293808435"
Nov 02 14:13:41.011801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:41.011259    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:41.011801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:41.011430    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 14:13:41.011801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:41.011489    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 14:13:41.045143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:41.045106    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:41.046403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:41.046381    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 14:13:41.046578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:41.046557    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 14:13:42.011290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:42.011155    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 14:13:42.011858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:42.011412    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 14:13:42.045380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:42.045350    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 14:13:42.045554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:42.045412    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 14:13:42.441434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:42.441355    5034 patch_prober.go:29] interesting pod/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:13:42.441434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:42.441403    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:13:43.008803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:43.008105    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 14:13:43.008803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:43.008167    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 14:13:43.044994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:43.044962    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 14:13:43.045406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:43.045037    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 14:13:44.012498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:44.010149    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 14:13:44.012498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:44.010210    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 14:13:44.051803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:44.050266    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 14:13:44.051803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:44.050349    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 14:13:44.469630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:44.465429    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podStartSLOduration=6.465409092 podStartE2EDuration="6.465409092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:40.646431115 +0000 UTC m=+1811.397735561" watchObservedRunningTime="2026-11-02 14:13:44.465409092 +0000 UTC m=+1815.216713530"
Nov 02 14:13:44.885803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:44.879475    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:44.941973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:44.920194    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"]
Nov 02 14:13:44.941973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:44.920194    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:13:44.977809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:44.974169    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:45.085803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.039877    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:45.085803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.040090    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 14:13:45.085803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.072116    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 14:13:45.085803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.081257    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 14:13:45.085803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.081314    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 14:13:45.245853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.245447    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:13:45.321803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.320393    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:45.325802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.325148    5034 generic.go:356] "Generic (PLEG): container finished" podID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerID="ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2" exitCode=0
Nov 02 14:13:45.325802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.325630    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" event={"ID":"60cfd239-3bd1-468f-81e1-bb85fee11656","Type":"ContainerDied","Data":"ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2"}
Nov 02 14:13:45.325802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.325705    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv" event={"ID":"60cfd239-3bd1-468f-81e1-bb85fee11656","Type":"ContainerDied","Data":"61b30fffcf56e089a51352aabe5c71bd4322bd800b4f7273ebcdea7e02f42bae"}
Nov 02 14:13:45.325802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.325749    5034 scope.go:122] "RemoveContainer" containerID="ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2"
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.357877    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.359065    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-token\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-token\") pod \"60cfd239-3bd1-468f-81e1-bb85fee11656\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") "
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.359187    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-certs\") pod \"60cfd239-3bd1-468f-81e1-bb85fee11656\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") "
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.359257    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-envoy\") pod \"60cfd239-3bd1-468f-81e1-bb85fee11656\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") "
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.359300    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-kube-api-access-99whh\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-kube-api-access-99whh\") pod \"60cfd239-3bd1-468f-81e1-bb85fee11656\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") "
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.359432    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-credential-socket\") pod \"60cfd239-3bd1-468f-81e1-bb85fee11656\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") "
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.359511    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-podinfo\") pod \"60cfd239-3bd1-468f-81e1-bb85fee11656\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") "
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.359562    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-socket\") pod \"60cfd239-3bd1-468f-81e1-bb85fee11656\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") "
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.359685    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-data\") pod \"60cfd239-3bd1-468f-81e1-bb85fee11656\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") "
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.359811    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60cfd239-3bd1-468f-81e1-bb85fee11656-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60cfd239-3bd1-468f-81e1-bb85fee11656-istiod-ca-cert\") pod \"60cfd239-3bd1-468f-81e1-bb85fee11656\" (UID: \"60cfd239-3bd1-468f-81e1-bb85fee11656\") "
Nov 02 14:13:45.365345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.361261    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-certs" pod "60cfd239-3bd1-468f-81e1-bb85fee11656" (UID: "60cfd239-3bd1-468f-81e1-bb85fee11656"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:45.370403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.370371    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-data" pod "60cfd239-3bd1-468f-81e1-bb85fee11656" (UID: "60cfd239-3bd1-468f-81e1-bb85fee11656"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:45.370699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.370676    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-socket" pod "60cfd239-3bd1-468f-81e1-bb85fee11656" (UID: "60cfd239-3bd1-468f-81e1-bb85fee11656"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:45.381892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.381533    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60cfd239-3bd1-468f-81e1-bb85fee11656-istiod-ca-cert" pod "60cfd239-3bd1-468f-81e1-bb85fee11656" (UID: "60cfd239-3bd1-468f-81e1-bb85fee11656"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:45.470803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.458164    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": context deadline exceeded"
Nov 02 14:13:45.501860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.477528    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:45.557806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.543416    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:45.557806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.543454    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:45.557806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.543476    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60cfd239-3bd1-468f-81e1-bb85fee11656-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:45.557806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.543490    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:45.579927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.578423    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-credential-socket" pod "60cfd239-3bd1-468f-81e1-bb85fee11656" (UID: "60cfd239-3bd1-468f-81e1-bb85fee11656"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:45.787871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.777592    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:45.787871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.778576    5034 scope.go:122] "RemoveContainer" containerID="9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308"
Nov 02 14:13:45.788131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.787899    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:13:45.809896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.788439    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-podinfo" pod "60cfd239-3bd1-468f-81e1-bb85fee11656" (UID: "60cfd239-3bd1-468f-81e1-bb85fee11656"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:13:45.809896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.788592    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-kube-api-access-99whh" pod "60cfd239-3bd1-468f-81e1-bb85fee11656" (UID: "60cfd239-3bd1-468f-81e1-bb85fee11656"). InnerVolumeSpecName "kube-api-access-99whh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:45.809896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.802824    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-envoy" pod "60cfd239-3bd1-468f-81e1-bb85fee11656" (UID: "60cfd239-3bd1-468f-81e1-bb85fee11656"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:13:45.821289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.821242    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-token" pod "60cfd239-3bd1-468f-81e1-bb85fee11656" (UID: "60cfd239-3bd1-468f-81e1-bb85fee11656"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:45.884049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.884011    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:45.884308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.884285    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99whh\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-kube-api-access-99whh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:45.884430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.884410    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:45.884539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.884526    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60cfd239-3bd1-468f-81e1-bb85fee11656-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:45.908950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.908880    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:45.935146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.935099    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:13:45.936219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.936194    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:13:45.959678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.959635    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:45.960409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.960386    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:13:45.973601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.973578    5034 scope.go:122] "RemoveContainer" containerID="ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2"
Nov 02 14:13:45.974426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.974393    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:45.980022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:45.979999    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:13:46.006187 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:13:46.005244    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2\": container with ID starting with ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2 not found: ID does not exist" containerID="ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2"
Nov 02 14:13:46.006423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.006395    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2"} err="failed to get container status \"ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2\": rpc error: code = NotFound desc = could not find container \"ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2\": container with ID starting with ea34dfd35608262bcaaca732784846d2054d118de48cbeb7545eded0eac992d2 not found: ID does not exist"
Nov 02 14:13:46.006555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.006540    5034 scope.go:122] "RemoveContainer" containerID="9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308"
Nov 02 14:13:46.007338 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:13:46.007313    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308\": container with ID starting with 9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308 not found: ID does not exist" containerID="9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308"
Nov 02 14:13:46.007592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.007569    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308"} err="failed to get container status \"9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308\": rpc error: code = NotFound desc = could not find container \"9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308\": container with ID starting with 9a5b917e657439f9d834460b9c9cf158be307e1fb69cf51c2a7ced3793e02308 not found: ID does not exist"
Nov 02 14:13:46.008132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.008085    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 14:13:46.008300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.008276    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 14:13:46.091227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.091130    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:46.092799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.092151    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:13:46.145801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.132205    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 14:13:46.145801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.132263    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 14:13:46.345858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.343737    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:13:46.345858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.344149    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:13:46.447065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.446993    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:13:46.486979 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.486937    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:46.487264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.487239    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 02 14:13:46.515521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.515488    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:13:46.539926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.539893    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:13:46.540153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.540141    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:13:46.816393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.816335    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:46.817342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.816746    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:13:46.820109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.820084    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc"
Nov 02 14:13:46.935887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.935842    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:46.953842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.937631    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:46.955670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.955606    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:13:46.964945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:46.964871    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:47.014146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.014111    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 14:13:47.014327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.014180    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 14:13:47.014569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.014545    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"
Nov 02 14:13:47.134072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.133954    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 14:13:47.134670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.134612    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 14:13:47.136107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.136081    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:47.302817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.271082    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:47.333292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.333258    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:13:47.431685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.431595    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:47.489273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.489214    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:13:47.513489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.513450    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:47.576956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.575879    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:47.837803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.831895    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:13:47.837803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.834632    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Nov 02 14:13:47.945919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.945757    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Nov 02 14:13:47.947912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.947818    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:47.968655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:47.968606    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" probeResult="failure" output="Get \"https://10.132.0.115:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:48.033252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.033215    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body=
Nov 02 14:13:48.054231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.054193    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused"
Nov 02 14:13:48.058653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.051951    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Nov 02 14:13:48.060092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.060062    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Nov 02 14:13:48.060295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.059242    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:13:48.077808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.077765    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:13:48.137889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.136922    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:13:48.266229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.266165    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"]
Nov 02 14:13:48.288110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.288037    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-8acd34-openshift-default-7895ccf77-7bptv"]
Nov 02 14:13:48.329742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.329713    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:13:48.332328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.332305    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:13:48.342955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.342933    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:13:48.399159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.399081    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:13:48.402204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.402184    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:13:48.406222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:48.406201    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:13:48.457300 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:13:48.457253    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:13:49.011580 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.011548    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:49.012070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.012047    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:49.012877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.012854    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:13:49.048347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.048323    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:49.048607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.048587    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:49.049328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.049307    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:13:49.396675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.396601    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" path="/var/lib/kubelet/pods/60cfd239-3bd1-468f-81e1-bb85fee11656/volumes"
Nov 02 14:13:49.535436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.535401    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"]
Nov 02 14:13:49.536052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.536037    5034 cpu_manager.go:414] "containerMap: removing container" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy"
Nov 02 14:13:49.536113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.536054    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy"
Nov 02 14:13:49.536113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.536064    5034 cpu_manager.go:414] "containerMap: removing container" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy"
Nov 02 14:13:49.536113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.536069    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy"
Nov 02 14:13:49.536216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.536193    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy"
Nov 02 14:13:49.536216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.536201    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="60cfd239-3bd1-468f-81e1-bb85fee11656" containerName="istio-proxy"
Nov 02 14:13:49.541309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.541287    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.556646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.556624    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d36463db-vvhlc\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.556763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.556679    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjvq4\" (UniqueName: \"kubernetes.io/projected/a1352c3e-b748-4236-a22e-6c42cf097997-kube-api-access-sjvq4\") pod \"maas-api-revoke-sub-d36463db-vvhlc\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.556831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.556766    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d36463db-vvhlc\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.559330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.559311    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:13:49.564209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.564190    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:13:49.564333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.564229    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:13:49.569545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.569527    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"]
Nov 02 14:13:49.658246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.658177    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d36463db-vvhlc\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.658246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.658238    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjvq4\" (UniqueName: \"kubernetes.io/projected/a1352c3e-b748-4236-a22e-6c42cf097997-kube-api-access-sjvq4\") pod \"maas-api-revoke-sub-d36463db-vvhlc\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.658426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.658268    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d36463db-vvhlc\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.659336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.659318    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:13:49.660662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.660636    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d36463db-vvhlc\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.669177 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.669154    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d36463db-vvhlc\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.683747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.683721    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjvq4\" (UniqueName: \"kubernetes.io/projected/a1352c3e-b748-4236-a22e-6c42cf097997-kube-api-access-sjvq4\") pod \"maas-api-revoke-sub-d36463db-vvhlc\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:49.856118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:49.856089    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:50.164700 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:50.164666    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1352c3e_b748_4236_a22e_6c42cf097997.slice/crio-7e907d0a7017bd102f0c0deeac1388e23f4a7e74855d633cfab351648d4fba3d WatchSource:0}: Error finding container 7e907d0a7017bd102f0c0deeac1388e23f4a7e74855d633cfab351648d4fba3d: Status 404 returned error can't find the container with id 7e907d0a7017bd102f0c0deeac1388e23f4a7e74855d633cfab351648d4fba3d
Nov 02 14:13:50.167349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:50.167326    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"]
Nov 02 14:13:50.326316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:50.326274    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj" podUID="8d0062d3-bf93-44ad-a827-85760d9cb85e" containerName="payload-pre-processing" containerID="cri-o://7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf" gracePeriod=25
Nov 02 14:13:50.582373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:50.582343    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:13:50.672713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:50.672686    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0062d3-bf93-44ad-a827-85760d9cb85e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0062d3-bf93-44ad-a827-85760d9cb85e-plugins-config-volume\") pod \"8d0062d3-bf93-44ad-a827-85760d9cb85e\" (UID: \"8d0062d3-bf93-44ad-a827-85760d9cb85e\") "
Nov 02 14:13:50.672913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:50.672893    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0062d3-bf93-44ad-a827-85760d9cb85e-kube-api-access-qxqzb\" (UniqueName: \"kubernetes.io/projected/8d0062d3-bf93-44ad-a827-85760d9cb85e-kube-api-access-qxqzb\") pod \"8d0062d3-bf93-44ad-a827-85760d9cb85e\" (UID: \"8d0062d3-bf93-44ad-a827-85760d9cb85e\") "
Nov 02 14:13:50.673182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:50.673140    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0062d3-bf93-44ad-a827-85760d9cb85e-plugins-config-volume" pod "8d0062d3-bf93-44ad-a827-85760d9cb85e" (UID: "8d0062d3-bf93-44ad-a827-85760d9cb85e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:50.673565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:50.673539    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d0062d3-bf93-44ad-a827-85760d9cb85e-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:50.675228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:50.675206    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0062d3-bf93-44ad-a827-85760d9cb85e-kube-api-access-qxqzb" pod "8d0062d3-bf93-44ad-a827-85760d9cb85e" (UID: "8d0062d3-bf93-44ad-a827-85760d9cb85e"). InnerVolumeSpecName "kube-api-access-qxqzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:50.774967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:50.774937    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxqzb\" (UniqueName: \"kubernetes.io/projected/8d0062d3-bf93-44ad-a827-85760d9cb85e-kube-api-access-qxqzb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:51.100065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.099629    5034 generic.go:356] "Generic (PLEG): container finished" podID="8d0062d3-bf93-44ad-a827-85760d9cb85e" containerID="7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf" exitCode=0
Nov 02 14:13:51.100065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.099674    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"
Nov 02 14:13:51.100065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.099886    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj" event={"ID":"8d0062d3-bf93-44ad-a827-85760d9cb85e","Type":"ContainerDied","Data":"7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf"}
Nov 02 14:13:51.100065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.099937    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj" event={"ID":"8d0062d3-bf93-44ad-a827-85760d9cb85e","Type":"ContainerDied","Data":"d8803f9c68004dd236f02d4c30f4cd858e1289569f7204892e3aca43f29c296b"}
Nov 02 14:13:51.100065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.099971    5034 scope.go:122] "RemoveContainer" containerID="7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf"
Nov 02 14:13:51.106893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.106852    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc" event={"ID":"a1352c3e-b748-4236-a22e-6c42cf097997","Type":"ContainerStarted","Data":"ea73fa8b4e9364a649445bdc3a680ae289bdd3b6badcf8a1169fe3333719b931"}
Nov 02 14:13:51.107188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.107165    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc" event={"ID":"a1352c3e-b748-4236-a22e-6c42cf097997","Type":"ContainerStarted","Data":"7e907d0a7017bd102f0c0deeac1388e23f4a7e74855d633cfab351648d4fba3d"}
Nov 02 14:13:51.129815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.129790    5034 scope.go:122] "RemoveContainer" containerID="7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf"
Nov 02 14:13:51.130135 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:13:51.130115    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf\": container with ID starting with 7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf not found: ID does not exist" containerID="7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf"
Nov 02 14:13:51.130205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.130147    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf"} err="failed to get container status \"7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf\": rpc error: code = NotFound desc = could not find container \"7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf\": container with ID starting with 7039ead204b74cae9a72eeb084a3df0d31c12ff2f161f3e1c98c9ceeb1d2a0cf not found: ID does not exist"
Nov 02 14:13:51.167824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.167759    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc" podStartSLOduration=2.167744537 podStartE2EDuration="2.167744537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:51.165728371 +0000 UTC m=+1821.917032809" watchObservedRunningTime="2026-11-02 14:13:51.167744537 +0000 UTC m=+1821.919048977"
Nov 02 14:13:51.218615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.218588    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"]
Nov 02 14:13:51.225100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.225075    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-732d3a-5c79cf9d7f-9p4lj"]
Nov 02 14:13:51.398199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.398123    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0062d3-bf93-44ad-a827-85760d9cb85e" path="/var/lib/kubelet/pods/8d0062d3-bf93-44ad-a827-85760d9cb85e/volumes"
Nov 02 14:13:51.500751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.500714    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"]
Nov 02 14:13:51.501743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.501723    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8d0062d3-bf93-44ad-a827-85760d9cb85e" containerName="payload-pre-processing"
Nov 02 14:13:51.501837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.501747    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0062d3-bf93-44ad-a827-85760d9cb85e" containerName="payload-pre-processing"
Nov 02 14:13:51.501941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.501928    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0062d3-bf93-44ad-a827-85760d9cb85e" containerName="payload-pre-processing"
Nov 02 14:13:51.506456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.506435    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.523719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.523693    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"]
Nov 02 14:13:51.602800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.599793    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-911e9272-s525t\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.602800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.599890    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-911e9272-s525t\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.602800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.599926    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkhw5\" (UniqueName: \"kubernetes.io/projected/348616d9-d3b9-438c-b932-57ebc5580d56-kube-api-access-mkhw5\") pod \"maas-api-revoke-sub-911e9272-s525t\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.701981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.701885    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-911e9272-s525t\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.702151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.701981    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-911e9272-s525t\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.702151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.702015    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkhw5\" (UniqueName: \"kubernetes.io/projected/348616d9-d3b9-438c-b932-57ebc5580d56-kube-api-access-mkhw5\") pod \"maas-api-revoke-sub-911e9272-s525t\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.702612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.702584    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-911e9272-s525t\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.705016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.704984    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-911e9272-s525t\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.727152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.727128    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkhw5\" (UniqueName: \"kubernetes.io/projected/348616d9-d3b9-438c-b932-57ebc5580d56-kube-api-access-mkhw5\") pod \"maas-api-revoke-sub-911e9272-s525t\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:51.823243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:51.823212    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:52.120900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:52.120869    5034 generic.go:356] "Generic (PLEG): container finished" podID="a1352c3e-b748-4236-a22e-6c42cf097997" containerID="ea73fa8b4e9364a649445bdc3a680ae289bdd3b6badcf8a1169fe3333719b931" exitCode=0
Nov 02 14:13:52.121053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:52.120952    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc" event={"ID":"a1352c3e-b748-4236-a22e-6c42cf097997","Type":"ContainerDied","Data":"ea73fa8b4e9364a649445bdc3a680ae289bdd3b6badcf8a1169fe3333719b931"}
Nov 02 14:13:52.142684 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:52.142653    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348616d9_d3b9_438c_b932_57ebc5580d56.slice/crio-ca957b89054f778aea3d6b08adf7127a4ab1867abb8e4d9613d2c48e4443c055 WatchSource:0}: Error finding container ca957b89054f778aea3d6b08adf7127a4ab1867abb8e4d9613d2c48e4443c055: Status 404 returned error can't find the container with id ca957b89054f778aea3d6b08adf7127a4ab1867abb8e4d9613d2c48e4443c055
Nov 02 14:13:52.145614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:52.145596    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"]
Nov 02 14:13:53.133485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.133455    5034 generic.go:356] "Generic (PLEG): container finished" podID="348616d9-d3b9-438c-b932-57ebc5580d56" containerID="3d9c5d05c9dd7a871edc8528934da0fe9bfc6b07247b346d2078f26edbd2f0de" exitCode=0
Nov 02 14:13:53.133902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.133536    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t" event={"ID":"348616d9-d3b9-438c-b932-57ebc5580d56","Type":"ContainerDied","Data":"3d9c5d05c9dd7a871edc8528934da0fe9bfc6b07247b346d2078f26edbd2f0de"}
Nov 02 14:13:53.133902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.133664    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t" event={"ID":"348616d9-d3b9-438c-b932-57ebc5580d56","Type":"ContainerStarted","Data":"ca957b89054f778aea3d6b08adf7127a4ab1867abb8e4d9613d2c48e4443c055"}
Nov 02 14:13:53.296280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.296243    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:53.428963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.428898    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1352c3e-b748-4236-a22e-6c42cf097997-kube-api-access-sjvq4\" (UniqueName: \"kubernetes.io/projected/a1352c3e-b748-4236-a22e-6c42cf097997-kube-api-access-sjvq4\") pod \"a1352c3e-b748-4236-a22e-6c42cf097997\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") "
Nov 02 14:13:53.428963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.428942    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-xks-ca\") pod \"a1352c3e-b748-4236-a22e-6c42cf097997\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") "
Nov 02 14:13:53.429175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.429107    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-openshift-ca\") pod \"a1352c3e-b748-4236-a22e-6c42cf097997\" (UID: \"a1352c3e-b748-4236-a22e-6c42cf097997\") "
Nov 02 14:13:53.429485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.429456    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-openshift-ca" pod "a1352c3e-b748-4236-a22e-6c42cf097997" (UID: "a1352c3e-b748-4236-a22e-6c42cf097997"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:53.429746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.429723    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:53.431190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.431164    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-xks-ca" pod "a1352c3e-b748-4236-a22e-6c42cf097997" (UID: "a1352c3e-b748-4236-a22e-6c42cf097997"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:53.431292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.431202    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1352c3e-b748-4236-a22e-6c42cf097997-kube-api-access-sjvq4" pod "a1352c3e-b748-4236-a22e-6c42cf097997" (UID: "a1352c3e-b748-4236-a22e-6c42cf097997"). InnerVolumeSpecName "kube-api-access-sjvq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:53.531064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.531030    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjvq4\" (UniqueName: \"kubernetes.io/projected/a1352c3e-b748-4236-a22e-6c42cf097997-kube-api-access-sjvq4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:53.531064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:53.531064    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1352c3e-b748-4236-a22e-6c42cf097997-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.144565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.144538    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"
Nov 02 14:13:54.145076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.144566    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc" event={"ID":"a1352c3e-b748-4236-a22e-6c42cf097997","Type":"ContainerDied","Data":"7e907d0a7017bd102f0c0deeac1388e23f4a7e74855d633cfab351648d4fba3d"}
Nov 02 14:13:54.145076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.144618    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e907d0a7017bd102f0c0deeac1388e23f4a7e74855d633cfab351648d4fba3d"
Nov 02 14:13:54.314635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.314608    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:54.445331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.445248    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-xks-ca\") pod \"348616d9-d3b9-438c-b932-57ebc5580d56\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") "
Nov 02 14:13:54.445502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.445390    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348616d9-d3b9-438c-b932-57ebc5580d56-kube-api-access-mkhw5\" (UniqueName: \"kubernetes.io/projected/348616d9-d3b9-438c-b932-57ebc5580d56-kube-api-access-mkhw5\") pod \"348616d9-d3b9-438c-b932-57ebc5580d56\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") "
Nov 02 14:13:54.445502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.445412    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-openshift-ca\") pod \"348616d9-d3b9-438c-b932-57ebc5580d56\" (UID: \"348616d9-d3b9-438c-b932-57ebc5580d56\") "
Nov 02 14:13:54.445845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.445817    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-openshift-ca" pod "348616d9-d3b9-438c-b932-57ebc5580d56" (UID: "348616d9-d3b9-438c-b932-57ebc5580d56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:13:54.447371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.447349    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-xks-ca" pod "348616d9-d3b9-438c-b932-57ebc5580d56" (UID: "348616d9-d3b9-438c-b932-57ebc5580d56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:13:54.447528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.447509    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348616d9-d3b9-438c-b932-57ebc5580d56-kube-api-access-mkhw5" pod "348616d9-d3b9-438c-b932-57ebc5580d56" (UID: "348616d9-d3b9-438c-b932-57ebc5580d56"). InnerVolumeSpecName "kube-api-access-mkhw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:13:54.546717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.546686    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkhw5\" (UniqueName: \"kubernetes.io/projected/348616d9-d3b9-438c-b932-57ebc5580d56-kube-api-access-mkhw5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.546717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.546712    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:54.546910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:54.546722    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348616d9-d3b9-438c-b932-57ebc5580d56-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:13:55.155215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.155186    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"
Nov 02 14:13:55.155604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.155218    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t" event={"ID":"348616d9-d3b9-438c-b932-57ebc5580d56","Type":"ContainerDied","Data":"ca957b89054f778aea3d6b08adf7127a4ab1867abb8e4d9613d2c48e4443c055"}
Nov 02 14:13:55.155604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.155266    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca957b89054f778aea3d6b08adf7127a4ab1867abb8e4d9613d2c48e4443c055"
Nov 02 14:13:55.257943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.257911    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"]
Nov 02 14:13:55.258929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.258907    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a1352c3e-b748-4236-a22e-6c42cf097997" containerName="revoke-keys"
Nov 02 14:13:55.259004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.258934    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1352c3e-b748-4236-a22e-6c42cf097997" containerName="revoke-keys"
Nov 02 14:13:55.259004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.258958    5034 cpu_manager.go:414] "containerMap: removing container" podUID="348616d9-d3b9-438c-b932-57ebc5580d56" containerName="revoke-keys"
Nov 02 14:13:55.259004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.258967    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348616d9-d3b9-438c-b932-57ebc5580d56" containerName="revoke-keys"
Nov 02 14:13:55.259166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.259152    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1352c3e-b748-4236-a22e-6c42cf097997" containerName="revoke-keys"
Nov 02 14:13:55.259203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.259172    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="348616d9-d3b9-438c-b932-57ebc5580d56" containerName="revoke-keys"
Nov 02 14:13:55.266822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.263839    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.293033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.293004    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"]
Nov 02 14:13:55.299301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.299280    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-2cddf2-openshift-default-dockercfg-8zz64\""
Nov 02 14:13:55.356502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.356475    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-envoy\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.356677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.356522    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kl9\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-kube-api-access-t7kl9\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.356750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.356670    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-podinfo\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.356750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.356722    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istiod-ca-cert\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.356900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.356809    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-certs\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.356900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.356845    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.356900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.356890    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-credential-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.357061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.356941    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-data\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.357061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.356981    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-token\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.458467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.458378    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-envoy\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.458467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.458437    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kl9\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-kube-api-access-t7kl9\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.458821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.458576    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-podinfo\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.458821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.458636    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istiod-ca-cert\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.458821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.458688    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-certs\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.458821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.458729    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.458821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.458805    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-credential-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.459153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.458843    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-data\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.459153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.458872    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-token\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.460711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.459440    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-certs\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.460711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.459547    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-credential-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.460711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.459751    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-data\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.461331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.461304    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.462026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.461982    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istiod-ca-cert\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.464380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.464359    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-envoy\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.468462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.468436    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-podinfo\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.483930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.483902    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-token\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.484128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.484107    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kl9\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-kube-api-access-t7kl9\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.595800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.593197    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:55.963717 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:13:55.963655    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d46d575_e8a7_4171_8a7a_6ecd59b16728.slice/crio-bc9f9ffe293d034bb372008a52c5417883380ee1aa18fa4236e435aa8f2e6fa9 WatchSource:0}: Error finding container bc9f9ffe293d034bb372008a52c5417883380ee1aa18fa4236e435aa8f2e6fa9: Status 404 returned error can't find the container with id bc9f9ffe293d034bb372008a52c5417883380ee1aa18fa4236e435aa8f2e6fa9
Nov 02 14:13:55.964860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.964837    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"]
Nov 02 14:13:55.966615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.966583    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:55.966688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.966647    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:55.966688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:55.966676    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:13:56.165927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:56.165879    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" event={"ID":"5d46d575-e8a7-4171-8a7a-6ecd59b16728","Type":"ContainerStarted","Data":"48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c"}
Nov 02 14:13:56.166362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:56.165970    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" event={"ID":"5d46d575-e8a7-4171-8a7a-6ecd59b16728","Type":"ContainerStarted","Data":"bc9f9ffe293d034bb372008a52c5417883380ee1aa18fa4236e435aa8f2e6fa9"}
Nov 02 14:13:56.219756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:56.219640    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" podStartSLOduration=1.219620551 podStartE2EDuration="1.219620551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:13:56.216103237 +0000 UTC m=+1826.967407675" watchObservedRunningTime="2026-11-02 14:13:56.219620551 +0000 UTC m=+1826.970924992"
Nov 02 14:13:56.593893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:56.593858    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:57.600509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:57.600479    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:58.181541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:58.181510    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:58.182575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:58.182553    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:13:59.921526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:59.921490    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"]
Nov 02 14:13:59.926182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:59.926164    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:13:59.946621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:59.946590    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"]
Nov 02 14:13:59.954740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:59.954714    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:13:59.954902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:59.954832    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:13:59.992016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:59.991989    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:13:59.992402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:13:59.992384    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:14:00.043190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.043159    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlq25\" (UniqueName: \"kubernetes.io/projected/bc162e6c-c868-4755-8acb-1be07abc3cbc-kube-api-access-xlq25\") pod \"maas-api-revoke-sub-d75cdf5a-vkfqq\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.043351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.043201    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d75cdf5a-vkfqq\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.043440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.043376    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d75cdf5a-vkfqq\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.145001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.144970    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlq25\" (UniqueName: \"kubernetes.io/projected/bc162e6c-c868-4755-8acb-1be07abc3cbc-kube-api-access-xlq25\") pod \"maas-api-revoke-sub-d75cdf5a-vkfqq\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.145202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.145014    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d75cdf5a-vkfqq\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.145202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.145128    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d75cdf5a-vkfqq\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.146013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.145987    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d75cdf5a-vkfqq\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.147663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.147639    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d75cdf5a-vkfqq\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.182305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.182250    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlq25\" (UniqueName: \"kubernetes.io/projected/bc162e6c-c868-4755-8acb-1be07abc3cbc-kube-api-access-xlq25\") pod \"maas-api-revoke-sub-d75cdf5a-vkfqq\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.244269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.244239    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:00.556026 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:00.555991    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc162e6c_c868_4755_8acb_1be07abc3cbc.slice/crio-e408a9a6793b48f987df05b22da735e5d7391bbecfee0fd77861a0a752f26acc WatchSource:0}: Error finding container e408a9a6793b48f987df05b22da735e5d7391bbecfee0fd77861a0a752f26acc: Status 404 returned error can't find the container with id e408a9a6793b48f987df05b22da735e5d7391bbecfee0fd77861a0a752f26acc
Nov 02 14:14:00.558952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:00.558920    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"]
Nov 02 14:14:01.206164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:01.206068    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq" event={"ID":"bc162e6c-c868-4755-8acb-1be07abc3cbc","Type":"ContainerStarted","Data":"0879910e666fb336013ca6e998458eb6241b6cf51573661a09a41edb5b5ba851"}
Nov 02 14:14:01.206164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:01.206134    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq" event={"ID":"bc162e6c-c868-4755-8acb-1be07abc3cbc","Type":"ContainerStarted","Data":"e408a9a6793b48f987df05b22da735e5d7391bbecfee0fd77861a0a752f26acc"}
Nov 02 14:14:01.258963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:01.258912    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq" podStartSLOduration=2.258897102 podStartE2EDuration="2.258897102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:01.257490797 +0000 UTC m=+1832.008795235" watchObservedRunningTime="2026-11-02 14:14:01.258897102 +0000 UTC m=+1832.010201540"
Nov 02 14:14:02.214909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:02.214873    5034 generic.go:356] "Generic (PLEG): container finished" podID="bc162e6c-c868-4755-8acb-1be07abc3cbc" containerID="0879910e666fb336013ca6e998458eb6241b6cf51573661a09a41edb5b5ba851" exitCode=0
Nov 02 14:14:02.215343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:02.214958    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq" event={"ID":"bc162e6c-c868-4755-8acb-1be07abc3cbc","Type":"ContainerDied","Data":"0879910e666fb336013ca6e998458eb6241b6cf51573661a09a41edb5b5ba851"}
Nov 02 14:14:03.129929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.116384    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"]
Nov 02 14:14:03.163413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.163351    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy" containerID="cri-o://07eccaca6e564c825b701e2b9ed9463bcf94d0f68139b7d3332defe280c23ed1" gracePeriod=30
Nov 02 14:14:03.583727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.583691    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:03.717161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.717122    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc162e6c-c868-4755-8acb-1be07abc3cbc-kube-api-access-xlq25\" (UniqueName: \"kubernetes.io/projected/bc162e6c-c868-4755-8acb-1be07abc3cbc-kube-api-access-xlq25\") pod \"bc162e6c-c868-4755-8acb-1be07abc3cbc\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") "
Nov 02 14:14:03.717505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.717478    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-xks-ca\") pod \"bc162e6c-c868-4755-8acb-1be07abc3cbc\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") "
Nov 02 14:14:03.717625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.717578    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-openshift-ca\") pod \"bc162e6c-c868-4755-8acb-1be07abc3cbc\" (UID: \"bc162e6c-c868-4755-8acb-1be07abc3cbc\") "
Nov 02 14:14:03.718706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.718642    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-openshift-ca" pod "bc162e6c-c868-4755-8acb-1be07abc3cbc" (UID: "bc162e6c-c868-4755-8acb-1be07abc3cbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:03.720036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.720010    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-xks-ca" pod "bc162e6c-c868-4755-8acb-1be07abc3cbc" (UID: "bc162e6c-c868-4755-8acb-1be07abc3cbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:14:03.720128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.720099    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc162e6c-c868-4755-8acb-1be07abc3cbc-kube-api-access-xlq25" pod "bc162e6c-c868-4755-8acb-1be07abc3cbc" (UID: "bc162e6c-c868-4755-8acb-1be07abc3cbc"). InnerVolumeSpecName "kube-api-access-xlq25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:03.819683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.819651    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlq25\" (UniqueName: \"kubernetes.io/projected/bc162e6c-c868-4755-8acb-1be07abc3cbc-kube-api-access-xlq25\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:03.819683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.819683    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:03.819910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:03.819696    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc162e6c-c868-4755-8acb-1be07abc3cbc-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:04.234045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:04.234018    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"
Nov 02 14:14:04.234224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:04.234049    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq" event={"ID":"bc162e6c-c868-4755-8acb-1be07abc3cbc","Type":"ContainerDied","Data":"e408a9a6793b48f987df05b22da735e5d7391bbecfee0fd77861a0a752f26acc"}
Nov 02 14:14:04.234224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:04.234098    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e408a9a6793b48f987df05b22da735e5d7391bbecfee0fd77861a0a752f26acc"
Nov 02 14:14:07.064607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.058856    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"]
Nov 02 14:14:07.064607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.060213    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bc162e6c-c868-4755-8acb-1be07abc3cbc" containerName="revoke-keys"
Nov 02 14:14:07.064607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.060247    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc162e6c-c868-4755-8acb-1be07abc3cbc" containerName="revoke-keys"
Nov 02 14:14:07.064607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.060491    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc162e6c-c868-4755-8acb-1be07abc3cbc" containerName="revoke-keys"
Nov 02 14:14:07.065549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.065506    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.067195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.067166    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"]
Nov 02 14:14:07.087821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.086063    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g22c\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-kube-api-access-9g22c\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.087821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.086181    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b383071-be95-4a6c-b9b4-805bebc7c15e-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.087821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.086218    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.087821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.086269    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.087821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.086359    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.087821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.086422    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.087821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.086487    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.087821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.086533    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.087821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.086610    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.091573 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.091301    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"]
Nov 02 14:14:07.091573 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.091428    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.205152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.205113    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.205820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.205774    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.207087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.207063    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.207299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.207266    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.207528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.207511    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.207725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.207695    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvmz8\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-kube-api-access-jvmz8\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.207905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.207878    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g22c\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-kube-api-access-9g22c\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.208180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.208145    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b383071-be95-4a6c-b9b4-805bebc7c15e-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.208351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.208331    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.208562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.208544    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.208840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.208818    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48deb00-e178-4318-a592-c53c7899bddf-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.209032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.209006    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.209188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.209171    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.209762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.209730    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.211026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.208883    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.211138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.208550    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.211568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.210578    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.214114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.213336    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.214349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.213881    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"]
Nov 02 14:14:07.214815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.214388    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.215487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.215373    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48deb00-e178-4318-a592-c53c7899bddf-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.215487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.215438    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.216416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.212301    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b383071-be95-4a6c-b9b4-805bebc7c15e-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.216416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.216363    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.219585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.219563    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.229571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.229541    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.256899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.256844    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g22c\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-kube-api-access-9g22c\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.257503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.257456    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.486762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.486732    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:07.513002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.511566    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48deb00-e178-4318-a592-c53c7899bddf-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.513002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.511626    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.513002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.511655    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.513002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.511701    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.513002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.511729    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.513002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.511765    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.513002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.511809    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48deb00-e178-4318-a592-c53c7899bddf-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.513002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.511920    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.513002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.512051    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvmz8\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-kube-api-access-jvmz8\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.514801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.514520    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-credential-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.515745 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.515593    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48deb00-e178-4318-a592-c53c7899bddf-istiod-ca-cert\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.516071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.516048    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-certs\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.536408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.535493    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48deb00-e178-4318-a592-c53c7899bddf-istio-podinfo\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.543523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.543008    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-data\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.543523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.543251    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-socket\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.552803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.551752    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-envoy\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.555340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.555317    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvmz8\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-kube-api-access-jvmz8\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.657078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.657005    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-istio-token\") pod \"e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") " pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:07.802201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:07.800972    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:08.682569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:08.682506    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:08.866669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:08.829972    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"]
Nov 02 14:14:08.866669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:08.830221    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy" containerID="cri-o://0f4c1650dd17b719470a10e82b6c8f86495a53db2b0352a2480472a30caecdff" gracePeriod=30
Nov 02 14:14:08.866669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:08.838863    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc" podUID="b539abab-f634-47a4-966f-7881e24cce86" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 02 14:14:09.129074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.128561    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:09.138930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.137897    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:09.235067 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:09.235011    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b383071_be95_4a6c_b9b4_805bebc7c15e.slice/crio-71be742b6bd68a08bf059e1c8a97dd5ddd2d9514488fb3790c37b25923ba2885 WatchSource:0}: Error finding container 71be742b6bd68a08bf059e1c8a97dd5ddd2d9514488fb3790c37b25923ba2885: Status 404 returned error can't find the container with id 71be742b6bd68a08bf059e1c8a97dd5ddd2d9514488fb3790c37b25923ba2885
Nov 02 14:14:09.243701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.240384    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:09.243701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.240512    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:09.243701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.240562    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:09.246133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.245328    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"]
Nov 02 14:14:09.354387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.354351    5034 generic.go:356] "Generic (PLEG): container finished" podID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerID="07eccaca6e564c825b701e2b9ed9463bcf94d0f68139b7d3332defe280c23ed1" exitCode=0
Nov 02 14:14:09.354550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.354530    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" event={"ID":"a7c3817b-5573-4131-8ec9-59a101e298f8","Type":"ContainerDied","Data":"07eccaca6e564c825b701e2b9ed9463bcf94d0f68139b7d3332defe280c23ed1"}
Nov 02 14:14:09.354619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.354591    5034 scope.go:122] "RemoveContainer" containerID="89990b0326ad6f86c2b4553aa75ae441ff81d5ef643f1e87b69ba8b9ef3c48d4"
Nov 02 14:14:09.356974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.356769    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" event={"ID":"4b383071-be95-4a6c-b9b4-805bebc7c15e","Type":"ContainerStarted","Data":"71be742b6bd68a08bf059e1c8a97dd5ddd2d9514488fb3790c37b25923ba2885"}
Nov 02 14:14:09.434556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.434496    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" podStartSLOduration=2.434478136 podStartE2EDuration="2.434478136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:09.424122619 +0000 UTC m=+1840.175427060" watchObservedRunningTime="2026-11-02 14:14:09.434478136 +0000 UTC m=+1840.185782576"
Nov 02 14:14:09.453330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.453287    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:14:09.491580 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.490117    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"]
Nov 02 14:14:09.491580 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:09.490677    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48deb00_e178_4318_a592_c53c7899bddf.slice/crio-0fab8dc7589b89fb6e9c98e058cdc602a1da8d6b403130bf0a8a97ed2a8c640a WatchSource:0}: Error finding container 0fab8dc7589b89fb6e9c98e058cdc602a1da8d6b403130bf0a8a97ed2a8c640a: Status 404 returned error can't find the container with id 0fab8dc7589b89fb6e9c98e058cdc602a1da8d6b403130bf0a8a97ed2a8c640a
Nov 02 14:14:09.493853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.493818    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:09.494111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.494086    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:09.494346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.494327    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:09.513038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.511298    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:09.563492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563203    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-socket\") pod \"a7c3817b-5573-4131-8ec9-59a101e298f8\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") "
Nov 02 14:14:09.563492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563314    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c3817b-5573-4131-8ec9-59a101e298f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c3817b-5573-4131-8ec9-59a101e298f8-istiod-ca-cert\") pod \"a7c3817b-5573-4131-8ec9-59a101e298f8\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") "
Nov 02 14:14:09.563492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563364    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-envoy\") pod \"a7c3817b-5573-4131-8ec9-59a101e298f8\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") "
Nov 02 14:14:09.563492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563405    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-socket" pod "a7c3817b-5573-4131-8ec9-59a101e298f8" (UID: "a7c3817b-5573-4131-8ec9-59a101e298f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:09.563492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563457    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-credential-socket\") pod \"a7c3817b-5573-4131-8ec9-59a101e298f8\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") "
Nov 02 14:14:09.563896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563589    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-kube-api-access-7pz9c\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-kube-api-access-7pz9c\") pod \"a7c3817b-5573-4131-8ec9-59a101e298f8\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") "
Nov 02 14:14:09.563896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563626    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-podinfo\") pod \"a7c3817b-5573-4131-8ec9-59a101e298f8\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") "
Nov 02 14:14:09.563896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563672    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-certs\") pod \"a7c3817b-5573-4131-8ec9-59a101e298f8\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") "
Nov 02 14:14:09.563896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563720    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-data\") pod \"a7c3817b-5573-4131-8ec9-59a101e298f8\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") "
Nov 02 14:14:09.563896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563721    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c3817b-5573-4131-8ec9-59a101e298f8-istiod-ca-cert" pod "a7c3817b-5573-4131-8ec9-59a101e298f8" (UID: "a7c3817b-5573-4131-8ec9-59a101e298f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:09.563896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.563829    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-token\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-token\") pod \"a7c3817b-5573-4131-8ec9-59a101e298f8\" (UID: \"a7c3817b-5573-4131-8ec9-59a101e298f8\") "
Nov 02 14:14:09.564916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.564888    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-credential-socket" pod "a7c3817b-5573-4131-8ec9-59a101e298f8" (UID: "a7c3817b-5573-4131-8ec9-59a101e298f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:09.565037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.564975    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-certs" pod "a7c3817b-5573-4131-8ec9-59a101e298f8" (UID: "a7c3817b-5573-4131-8ec9-59a101e298f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:09.565165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.565116    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:09.565165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.565135    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:09.565165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.565139    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-data" pod "a7c3817b-5573-4131-8ec9-59a101e298f8" (UID: "a7c3817b-5573-4131-8ec9-59a101e298f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:09.565165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.565150    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7c3817b-5573-4131-8ec9-59a101e298f8-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:09.565165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.565164    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:09.566621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.566577    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-envoy" pod "a7c3817b-5573-4131-8ec9-59a101e298f8" (UID: "a7c3817b-5573-4131-8ec9-59a101e298f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:09.567521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.567494    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-podinfo" pod "a7c3817b-5573-4131-8ec9-59a101e298f8" (UID: "a7c3817b-5573-4131-8ec9-59a101e298f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:09.568211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.568185    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-token" pod "a7c3817b-5573-4131-8ec9-59a101e298f8" (UID: "a7c3817b-5573-4131-8ec9-59a101e298f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:09.575833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.573401    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-kube-api-access-7pz9c" pod "a7c3817b-5573-4131-8ec9-59a101e298f8" (UID: "a7c3817b-5573-4131-8ec9-59a101e298f8"). InnerVolumeSpecName "kube-api-access-7pz9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:09.666533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.666495    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pz9c\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-kube-api-access-7pz9c\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:09.666533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.666535    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:09.666533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.666550    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:09.666891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.666562    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:09.666891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:09.666577    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7c3817b-5573-4131-8ec9-59a101e298f8-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:10.372859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.368950    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d" event={"ID":"a7c3817b-5573-4131-8ec9-59a101e298f8","Type":"ContainerDied","Data":"a16a6ab70b1496dc2b1dbec8878728c3b3d307df54f328ff8e9b7f5e3c3eb16a"}
Nov 02 14:14:10.372859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.369056    5034 scope.go:122] "RemoveContainer" containerID="07eccaca6e564c825b701e2b9ed9463bcf94d0f68139b7d3332defe280c23ed1"
Nov 02 14:14:10.372859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.369227    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"
Nov 02 14:14:10.429820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.429447    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" event={"ID":"4b383071-be95-4a6c-b9b4-805bebc7c15e","Type":"ContainerStarted","Data":"87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d"}
Nov 02 14:14:10.433200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.433099    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" event={"ID":"b48deb00-e178-4318-a592-c53c7899bddf","Type":"ContainerStarted","Data":"f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9"}
Nov 02 14:14:10.433809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.433141    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" event={"ID":"b48deb00-e178-4318-a592-c53c7899bddf","Type":"ContainerStarted","Data":"0fab8dc7589b89fb6e9c98e058cdc602a1da8d6b403130bf0a8a97ed2a8c640a"}
Nov 02 14:14:10.516381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.516346    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 14:14:10.516675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.516648    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 14:14:10.522809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.519561    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" podStartSLOduration=3.519542908 podStartE2EDuration="3.519542908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:10.492227799 +0000 UTC m=+1841.243532238" watchObservedRunningTime="2026-11-02 14:14:10.519542908 +0000 UTC m=+1841.270847346"
Nov 02 14:14:10.555853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.553891    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"]
Nov 02 14:14:10.578802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.560312    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-732d3a-openshift-default-cd9f4b85-2682d"]
Nov 02 14:14:10.807861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.803217    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:10.831822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.829034    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 14:14:10.831822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:10.829094    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 14:14:11.038842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.038801    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"]
Nov 02 14:14:11.047807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.040104    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy"
Nov 02 14:14:11.047807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.046216    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy"
Nov 02 14:14:11.047807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.046251    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy"
Nov 02 14:14:11.047807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.046262    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy"
Nov 02 14:14:11.047807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.046593    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy"
Nov 02 14:14:11.047807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.046612    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" containerName="istio-proxy"
Nov 02 14:14:11.052499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.052475    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"]
Nov 02 14:14:11.052677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.052595    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:11.077126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.077097    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-4b7759\""
Nov 02 14:14:11.094865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.078346    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-4b7759-dockercfg-qdxrs\""
Nov 02 14:14:11.171800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.170660    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pll2m\" (UniqueName: \"kubernetes.io/projected/360e6c56-ada6-44b4-b491-dfb228dea4ad-kube-api-access-pll2m\") pod \"payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9\" (UID: \"360e6c56-ada6-44b4-b491-dfb228dea4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:11.171800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.170853    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360e6c56-ada6-44b4-b491-dfb228dea4ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9\" (UID: \"360e6c56-ada6-44b4-b491-dfb228dea4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:11.288801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.278560    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360e6c56-ada6-44b4-b491-dfb228dea4ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9\" (UID: \"360e6c56-ada6-44b4-b491-dfb228dea4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:11.288801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.278831    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pll2m\" (UniqueName: \"kubernetes.io/projected/360e6c56-ada6-44b4-b491-dfb228dea4ad-kube-api-access-pll2m\") pod \"payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9\" (UID: \"360e6c56-ada6-44b4-b491-dfb228dea4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:11.288801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.280749    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360e6c56-ada6-44b4-b491-dfb228dea4ad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9\" (UID: \"360e6c56-ada6-44b4-b491-dfb228dea4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:11.504849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.493497    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pll2m\" (UniqueName: \"kubernetes.io/projected/360e6c56-ada6-44b4-b491-dfb228dea4ad-kube-api-access-pll2m\") pod \"payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9\" (UID: \"360e6c56-ada6-44b4-b491-dfb228dea4ad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:11.526853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.510889    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 14:14:11.526853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.510942    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 14:14:11.552461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.552427    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c3817b-5573-4131-8ec9-59a101e298f8" path="/var/lib/kubelet/pods/a7c3817b-5573-4131-8ec9-59a101e298f8/volumes"
Nov 02 14:14:11.554380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.554349    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"]
Nov 02 14:14:11.564258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.564232    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"]
Nov 02 14:14:11.564382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.564338    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:11.651461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.651434    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dghx6\" (UniqueName: \"kubernetes.io/projected/f55f848e-281e-45a4-bb5f-77fea43cd292-kube-api-access-dghx6\") pod \"payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z\" (UID: \"f55f848e-281e-45a4-bb5f-77fea43cd292\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:11.651791 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.651764    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f848e-281e-45a4-bb5f-77fea43cd292-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z\" (UID: \"f55f848e-281e-45a4-bb5f-77fea43cd292\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:11.753138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.753096    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f848e-281e-45a4-bb5f-77fea43cd292-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z\" (UID: \"f55f848e-281e-45a4-bb5f-77fea43cd292\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:11.753711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.753683    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dghx6\" (UniqueName: \"kubernetes.io/projected/f55f848e-281e-45a4-bb5f-77fea43cd292-kube-api-access-dghx6\") pod \"payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z\" (UID: \"f55f848e-281e-45a4-bb5f-77fea43cd292\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:11.754016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.753993    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f848e-281e-45a4-bb5f-77fea43cd292-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z\" (UID: \"f55f848e-281e-45a4-bb5f-77fea43cd292\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:11.756914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.756837    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:11.799751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.799688    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dghx6\" (UniqueName: \"kubernetes.io/projected/f55f848e-281e-45a4-bb5f-77fea43cd292-kube-api-access-dghx6\") pod \"payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z\" (UID: \"f55f848e-281e-45a4-bb5f-77fea43cd292\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:11.804217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.804189    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 14:14:11.804520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.804401    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 14:14:11.957014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:11.953646    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:12.454245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.453815    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"]
Nov 02 14:14:12.510297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.510264    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 14:14:12.511120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.510813    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 14:14:12.550534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.550496    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"]
Nov 02 14:14:12.654438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.654349    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9" event={"ID":"360e6c56-ada6-44b4-b491-dfb228dea4ad","Type":"ContainerStarted","Data":"cde78c2a03fefa5815e28707c6843d35b66f4b099c33d36523e02bc596d2a1e2"}
Nov 02 14:14:12.659166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.659068    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z" event={"ID":"f55f848e-281e-45a4-bb5f-77fea43cd292","Type":"ContainerStarted","Data":"6a116ed4e9fb53636d5d1bfde73b85534c0b41dee320a4bdfe0487eedada7bf7"}
Nov 02 14:14:12.811803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.805328    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 14:14:12.811803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.805404    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 14:14:12.860481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.856350    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"]
Nov 02 14:14:12.862017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.861992    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:12.891805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.889048    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-960a68\""
Nov 02 14:14:12.891805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.889191    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-960a68-dockercfg-6dgch\""
Nov 02 14:14:12.891805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.890908    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf25039-006c-447b-bbc7-a418d169eb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq\" (UID: \"eaf25039-006c-447b-bbc7-a418d169eb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:12.891805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:12.890997    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c9qn\" (UniqueName: \"kubernetes.io/projected/eaf25039-006c-447b-bbc7-a418d169eb39-kube-api-access-9c9qn\") pod \"payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq\" (UID: \"eaf25039-006c-447b-bbc7-a418d169eb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:13.120814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.113378    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf25039-006c-447b-bbc7-a418d169eb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq\" (UID: \"eaf25039-006c-447b-bbc7-a418d169eb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:13.187828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.161678    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf25039-006c-447b-bbc7-a418d169eb39-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq\" (UID: \"eaf25039-006c-447b-bbc7-a418d169eb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:13.187828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.161869    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c9qn\" (UniqueName: \"kubernetes.io/projected/eaf25039-006c-447b-bbc7-a418d169eb39-kube-api-access-9c9qn\") pod \"payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq\" (UID: \"eaf25039-006c-447b-bbc7-a418d169eb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:13.248160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.247905    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"]
Nov 02 14:14:13.456542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.454637    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c9qn\" (UniqueName: \"kubernetes.io/projected/eaf25039-006c-447b-bbc7-a418d169eb39-kube-api-access-9c9qn\") pod \"payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq\" (UID: \"eaf25039-006c-447b-bbc7-a418d169eb39\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:13.468215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.468187    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"]
Nov 02 14:14:13.553822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.553598    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 14:14:13.553822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.553668    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 14:14:13.554368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.554299    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"]
Nov 02 14:14:13.554451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.554441    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:13.554514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.554495    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:13.631863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.613524    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstvm\" (UniqueName: \"kubernetes.io/projected/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-kube-api-access-wstvm\") pod \"payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc\" (UID: \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:13.641833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.640996    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc\" (UID: \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:13.680227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.678966    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z" event={"ID":"f55f848e-281e-45a4-bb5f-77fea43cd292","Type":"ContainerStarted","Data":"699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf"}
Nov 02 14:14:13.680227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.680144    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:13.690810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.683234    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9" event={"ID":"360e6c56-ada6-44b4-b491-dfb228dea4ad","Type":"ContainerStarted","Data":"c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562"}
Nov 02 14:14:13.691089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.691009    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:13.749820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.745478    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc\" (UID: \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:13.749820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.745839    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstvm\" (UniqueName: \"kubernetes.io/projected/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-kube-api-access-wstvm\") pod \"payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc\" (UID: \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:13.750433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.750395    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc\" (UID: \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:13.851162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.847323    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 14:14:13.851162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.847385    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 14:14:13.858443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.854048    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z" podStartSLOduration=2.8540291509999998 podStartE2EDuration="2.854029151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:13.736090837 +0000 UTC m=+1844.487395276" watchObservedRunningTime="2026-11-02 14:14:13.854029151 +0000 UTC m=+1844.605333594"
Nov 02 14:14:13.871806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.869071    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstvm\" (UniqueName: \"kubernetes.io/projected/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-kube-api-access-wstvm\") pod \"payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc\" (UID: \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:13.946802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:13.946368    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:14.264078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.263654    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9" podStartSLOduration=4.263632235 podStartE2EDuration="4.263632235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:13.846060035 +0000 UTC m=+1844.597364497" watchObservedRunningTime="2026-11-02 14:14:14.263632235 +0000 UTC m=+1845.014936670"
Nov 02 14:14:14.265066 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:14.264993    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf25039_006c_447b_bbc7_a418d169eb39.slice/crio-568fd54eb56d702e3f4f837667be11be5299462da048e1f38e2e08fc94231c94 WatchSource:0}: Error finding container 568fd54eb56d702e3f4f837667be11be5299462da048e1f38e2e08fc94231c94: Status 404 returned error can't find the container with id 568fd54eb56d702e3f4f837667be11be5299462da048e1f38e2e08fc94231c94
Nov 02 14:14:14.297852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.274625    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"]
Nov 02 14:14:14.536858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.536742    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 14:14:14.537133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.537092    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 14:14:14.852263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.847089    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 14:14:14.852263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.847147    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 14:14:14.889799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.883535    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"]
Nov 02 14:14:14.950065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.935247    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq" event={"ID":"eaf25039-006c-447b-bbc7-a418d169eb39","Type":"ContainerStarted","Data":"7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81"}
Nov 02 14:14:14.950065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.935325    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq" event={"ID":"eaf25039-006c-447b-bbc7-a418d169eb39","Type":"ContainerStarted","Data":"568fd54eb56d702e3f4f837667be11be5299462da048e1f38e2e08fc94231c94"}
Nov 02 14:14:14.950065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.949947    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:14.985747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.984075    5034 generic.go:356] "Generic (PLEG): container finished" podID="b69662c7-cd64-4242-84fb-bd6deea012be" containerID="0f4c1650dd17b719470a10e82b6c8f86495a53db2b0352a2480472a30caecdff" exitCode=0
Nov 02 14:14:14.985747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.984942    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" event={"ID":"b69662c7-cd64-4242-84fb-bd6deea012be","Type":"ContainerDied","Data":"0f4c1650dd17b719470a10e82b6c8f86495a53db2b0352a2480472a30caecdff"}
Nov 02 14:14:14.985747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:14.984999    5034 scope.go:122] "RemoveContainer" containerID="c59ae6cb7f477809dfeb513f2aeaf5ac4b80849419bae81064b12d394e1c6c40"
Nov 02 14:14:15.038817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.035903    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq" podStartSLOduration=3.035884476 podStartE2EDuration="3.035884476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:15.02644403 +0000 UTC m=+1845.777748469" watchObservedRunningTime="2026-11-02 14:14:15.035884476 +0000 UTC m=+1845.787188916"
Nov 02 14:14:15.086587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.086508    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:14:15.172466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.172435    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-credential-socket\") pod \"b69662c7-cd64-4242-84fb-bd6deea012be\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") "
Nov 02 14:14:15.187826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.172831    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b69662c7-cd64-4242-84fb-bd6deea012be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69662c7-cd64-4242-84fb-bd6deea012be-istiod-ca-cert\") pod \"b69662c7-cd64-4242-84fb-bd6deea012be\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") "
Nov 02 14:14:15.187826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.172921    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-kube-api-access-mj49m\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-kube-api-access-mj49m\") pod \"b69662c7-cd64-4242-84fb-bd6deea012be\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") "
Nov 02 14:14:15.187826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.172978    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-data\") pod \"b69662c7-cd64-4242-84fb-bd6deea012be\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") "
Nov 02 14:14:15.187826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.173118    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b69662c7-cd64-4242-84fb-bd6deea012be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69662c7-cd64-4242-84fb-bd6deea012be-istio-podinfo\") pod \"b69662c7-cd64-4242-84fb-bd6deea012be\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") "
Nov 02 14:14:15.187826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.173163    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-istio-token\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-istio-token\") pod \"b69662c7-cd64-4242-84fb-bd6deea012be\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") "
Nov 02 14:14:15.187826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.173265    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-envoy\") pod \"b69662c7-cd64-4242-84fb-bd6deea012be\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") "
Nov 02 14:14:15.187826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.173307    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-socket\") pod \"b69662c7-cd64-4242-84fb-bd6deea012be\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") "
Nov 02 14:14:15.187826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.173409    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-certs\") pod \"b69662c7-cd64-4242-84fb-bd6deea012be\" (UID: \"b69662c7-cd64-4242-84fb-bd6deea012be\") "
Nov 02 14:14:15.187826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.186828    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-credential-socket" pod "b69662c7-cd64-4242-84fb-bd6deea012be" (UID: "b69662c7-cd64-4242-84fb-bd6deea012be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:15.188469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.188368    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-socket" pod "b69662c7-cd64-4242-84fb-bd6deea012be" (UID: "b69662c7-cd64-4242-84fb-bd6deea012be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:15.188622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.188597    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-data" pod "b69662c7-cd64-4242-84fb-bd6deea012be" (UID: "b69662c7-cd64-4242-84fb-bd6deea012be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:15.190407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.188791    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-certs" pod "b69662c7-cd64-4242-84fb-bd6deea012be" (UID: "b69662c7-cd64-4242-84fb-bd6deea012be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:15.220972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.220899    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69662c7-cd64-4242-84fb-bd6deea012be-istiod-ca-cert" pod "b69662c7-cd64-4242-84fb-bd6deea012be" (UID: "b69662c7-cd64-4242-84fb-bd6deea012be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:15.293114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.284453    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.293114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.284495    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.293114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.284511    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.293114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.284525    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.293114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.284541    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b69662c7-cd64-4242-84fb-bd6deea012be-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.345415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.343879    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-istio-token" pod "b69662c7-cd64-4242-84fb-bd6deea012be" (UID: "b69662c7-cd64-4242-84fb-bd6deea012be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:15.346978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.346936    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-kube-api-access-mj49m" pod "b69662c7-cd64-4242-84fb-bd6deea012be" (UID: "b69662c7-cd64-4242-84fb-bd6deea012be"). InnerVolumeSpecName "kube-api-access-mj49m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:15.350607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.350563    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b69662c7-cd64-4242-84fb-bd6deea012be-istio-podinfo" pod "b69662c7-cd64-4242-84fb-bd6deea012be" (UID: "b69662c7-cd64-4242-84fb-bd6deea012be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:15.351138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.351113    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-envoy" pod "b69662c7-cd64-4242-84fb-bd6deea012be" (UID: "b69662c7-cd64-4242-84fb-bd6deea012be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:15.436084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.394577    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b69662c7-cd64-4242-84fb-bd6deea012be-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.436084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.394612    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.436084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.394626    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b69662c7-cd64-4242-84fb-bd6deea012be-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.436084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.394639    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj49m\" (UniqueName: \"kubernetes.io/projected/b69662c7-cd64-4242-84fb-bd6deea012be-kube-api-access-mj49m\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:15.514977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.514925    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 14:14:15.515286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.515244    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 14:14:15.806804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.806150    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 14:14:15.806804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:15.806215    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 14:14:16.014065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.014032    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"
Nov 02 14:14:16.014560 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.014041    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv" event={"ID":"b69662c7-cd64-4242-84fb-bd6deea012be","Type":"ContainerDied","Data":"20918ef7a658a2017d45f4a152c95cebe7b9dcb8b8ac79c006e76afa6ac9569b"}
Nov 02 14:14:16.014560 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.014178    5034 scope.go:122] "RemoveContainer" containerID="0f4c1650dd17b719470a10e82b6c8f86495a53db2b0352a2480472a30caecdff"
Nov 02 14:14:16.019489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.019464    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc" event={"ID":"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc","Type":"ContainerStarted","Data":"5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22"}
Nov 02 14:14:16.019606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.019513    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:16.019606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.019540    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc" event={"ID":"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc","Type":"ContainerStarted","Data":"42f7a93aa7735ab317912162409393c995010a326626792ca4f1c035180cfae2"}
Nov 02 14:14:16.117134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.117095    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"]
Nov 02 14:14:16.125194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.125161    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c07f7e-openshift-default-6f855bb76f-znjlv"]
Nov 02 14:14:16.162003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.161957    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc" podStartSLOduration=4.161942658 podStartE2EDuration="4.161942658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:16.160384174 +0000 UTC m=+1846.911688612" watchObservedRunningTime="2026-11-02 14:14:16.161942658 +0000 UTC m=+1846.913247096"
Nov 02 14:14:16.478469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.478378    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"]
Nov 02 14:14:16.479747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.479716    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy"
Nov 02 14:14:16.479747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.479748    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy"
Nov 02 14:14:16.480024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.480007    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy"
Nov 02 14:14:16.480090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.480037    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy"
Nov 02 14:14:16.486148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.486120    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.502678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.502634    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:14:16.502812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.502716    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:14:16.503281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.503165    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:14:16.503281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.503250    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:14:16.506669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.506640    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"]
Nov 02 14:14:16.511645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.510749    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 02 14:14:16.511645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.510818    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 02 14:14:16.527224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.527198    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-979ml\" (UniqueName: \"kubernetes.io/projected/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-kube-api-access-979ml\") pod \"maas-api-revoke-sub-547482f3-jbrgv\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.527473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.527447    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-547482f3-jbrgv\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.527526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.527508    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-547482f3-jbrgv\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.629546 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.629506    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-979ml\" (UniqueName: \"kubernetes.io/projected/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-kube-api-access-979ml\") pod \"maas-api-revoke-sub-547482f3-jbrgv\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.629994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.629968    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-547482f3-jbrgv\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.630200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.630179    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-547482f3-jbrgv\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.631364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.631342    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-547482f3-jbrgv\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.642242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.642209    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-547482f3-jbrgv\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.659336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.659281    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-979ml\" (UniqueName: \"kubernetes.io/projected/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-kube-api-access-979ml\") pod \"maas-api-revoke-sub-547482f3-jbrgv\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.719464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.719427    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"]
Nov 02 14:14:16.720212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.720191    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy"
Nov 02 14:14:16.720324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.720212    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" containerName="istio-proxy"
Nov 02 14:14:16.724451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.724427    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.767762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.767680    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"]
Nov 02 14:14:16.775577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.775549    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"]
Nov 02 14:14:16.794442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.794416    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"]
Nov 02 14:14:16.794767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.794693    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.802841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.801899    5034 patch_prober.go:29] interesting pod/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 02 14:14:16.802841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.801950    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 02 14:14:16.813882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.813857    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:16.834211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834181    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.834407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834310    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.834407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834355    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.834525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834411    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.834525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834445    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.834525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834500    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.834683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834536    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72df336e-41ed-45c2-b132-7c715685584e-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.834683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834600    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.834683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834655    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2n68\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-kube-api-access-r2n68\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.834856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834687    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.834856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834714    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.834856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834747    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.835012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.834928    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72df336e-41ed-45c2-b132-7c715685584e-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.835081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.835013    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5fj\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-kube-api-access-lm5fj\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.835555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.835214    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.835555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.835310    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.835555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.835374    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.835555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.835414    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.937801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937488    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.937801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937539    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.937801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937581    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.937801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937615    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.937801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937665    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.937801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937689    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72df336e-41ed-45c2-b132-7c715685584e-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.937801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937738    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.937801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937803    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2n68\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-kube-api-access-r2n68\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937833    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937858    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937928    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937975    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72df336e-41ed-45c2-b132-7c715685584e-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.937995    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5fj\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-kube-api-access-lm5fj\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.938021    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.938054    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.938091    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.938124    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.938405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.938179    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.939013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.938861    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.939013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.938982    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.940324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.939246    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.940324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.939716    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.940324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.939949    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72df336e-41ed-45c2-b132-7c715685584e-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.940324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.940087    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-certs\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.940324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.940206    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.940324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.940267    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-credential-socket\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.940863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.940841    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istiod-ca-cert\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.941012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.940984    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-data\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.944695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.944131    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.945183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.945161    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72df336e-41ed-45c2-b132-7c715685584e-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.954091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.954039    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-envoy\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.967614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.967570    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-podinfo\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.984117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.983731    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:16.985396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.985328    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-istio-token\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.986012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.985963    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5fj\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-kube-api-access-lm5fj\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:16.986306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:16.986260    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2n68\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-kube-api-access-r2n68\") pod \"e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") " pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:17.066481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.066381    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:17.125689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.123134    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:17.166583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.166549    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"]
Nov 02 14:14:17.406169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.405771    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b69662c7-cd64-4242-84fb-bd6deea012be" path="/var/lib/kubelet/pods/b69662c7-cd64-4242-84fb-bd6deea012be/volumes"
Nov 02 14:14:17.493611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.493584    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"]
Nov 02 14:14:17.494409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.494379    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:17.494475 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.494452    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:17.494515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.494491    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:17.557941 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:17.557908    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fba0a38_2bbe_41ee_a8a6_8a7b8d7089eb.slice/crio-9434c766463df280499193490969ca8d186f4aa8503e94596bc4fc49e686a935 WatchSource:0}: Error finding container 9434c766463df280499193490969ca8d186f4aa8503e94596bc4fc49e686a935: Status 404 returned error can't find the container with id 9434c766463df280499193490969ca8d186f4aa8503e94596bc4fc49e686a935
Nov 02 14:14:17.558520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.558498    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"]
Nov 02 14:14:17.561201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.561173    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:17.561294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.561249    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:17.561357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.561294    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:17.592852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.591847    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:17.594169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.593581    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:17.595467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.595413    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:14:17.975935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.975854    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:17.977205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.977069    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:17.978869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:17.978827    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:14:18.097965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.097924    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv" event={"ID":"f89ab83e-c555-4652-9cc4-9d43af3bb0c3","Type":"ContainerStarted","Data":"2c1c4475cc3eedee7daf6febba775ebf1b68216214cbff5f08452981bb09758d"}
Nov 02 14:14:18.098594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.098570    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv" event={"ID":"f89ab83e-c555-4652-9cc4-9d43af3bb0c3","Type":"ContainerStarted","Data":"1d62bc918dc7adfbe5718a1c59f376a7ddffa0c4c8e94cb5db3e99ed95642b1c"}
Nov 02 14:14:18.105160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.104358    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" event={"ID":"72df336e-41ed-45c2-b132-7c715685584e","Type":"ContainerStarted","Data":"bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273"}
Nov 02 14:14:18.105160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.104414    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" event={"ID":"72df336e-41ed-45c2-b132-7c715685584e","Type":"ContainerStarted","Data":"bc6b19dd90d21d5d5f4e37a44496e2e7d44917001592c43faa34648252e72bdc"}
Nov 02 14:14:18.120761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.120729    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" event={"ID":"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb","Type":"ContainerStarted","Data":"1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5"}
Nov 02 14:14:18.122147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.122127    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" event={"ID":"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb","Type":"ContainerStarted","Data":"9434c766463df280499193490969ca8d186f4aa8503e94596bc4fc49e686a935"}
Nov 02 14:14:18.125188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.125172    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:18.125377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.125361    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 14:14:18.125495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.125477    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 14:14:18.155173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.155107    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv" podStartSLOduration=2.155088247 podStartE2EDuration="2.155088247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:18.151581139 +0000 UTC m=+1848.902885578" watchObservedRunningTime="2026-11-02 14:14:18.155088247 +0000 UTC m=+1848.906392686"
Nov 02 14:14:18.224950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.223219    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" podStartSLOduration=2.223199009 podStartE2EDuration="2.223199009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:18.202289555 +0000 UTC m=+1848.953593993" watchObservedRunningTime="2026-11-02 14:14:18.223199009 +0000 UTC m=+1848.974503448"
Nov 02 14:14:18.269118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:18.269050    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" podStartSLOduration=2.269028861 podStartE2EDuration="2.269028861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:18.262868357 +0000 UTC m=+1849.014172789" watchObservedRunningTime="2026-11-02 14:14:18.269028861 +0000 UTC m=+1849.020333300"
Nov 02 14:14:18.651293 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:14:18.651207    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:14:19.068138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:19.068109    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:19.091799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:19.089114    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 14:14:19.091799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:19.089180    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 14:14:19.123767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:19.123724    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 14:14:19.124397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:19.124364    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 14:14:19.140817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:19.139325    5034 generic.go:356] "Generic (PLEG): container finished" podID="f89ab83e-c555-4652-9cc4-9d43af3bb0c3" containerID="2c1c4475cc3eedee7daf6febba775ebf1b68216214cbff5f08452981bb09758d" exitCode=0
Nov 02 14:14:19.140817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:19.139610    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv" event={"ID":"f89ab83e-c555-4652-9cc4-9d43af3bb0c3","Type":"ContainerDied","Data":"2c1c4475cc3eedee7daf6febba775ebf1b68216214cbff5f08452981bb09758d"}
Nov 02 14:14:20.068046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.068011    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 14:14:20.068230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.068069    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 14:14:20.130828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.129919    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 14:14:20.130828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.129981    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 14:14:20.651447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.651421    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:20.762799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.762235    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-kube-api-access-979ml\" (UniqueName: \"kubernetes.io/projected/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-kube-api-access-979ml\") pod \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") "
Nov 02 14:14:20.762799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.762306    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-openshift-ca\") pod \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") "
Nov 02 14:14:20.762799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.762367    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-xks-ca\") pod \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\" (UID: \"f89ab83e-c555-4652-9cc4-9d43af3bb0c3\") "
Nov 02 14:14:20.762799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.762667    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-openshift-ca" pod "f89ab83e-c555-4652-9cc4-9d43af3bb0c3" (UID: "f89ab83e-c555-4652-9cc4-9d43af3bb0c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:20.763292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.763270    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:20.764558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.764533    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-xks-ca" pod "f89ab83e-c555-4652-9cc4-9d43af3bb0c3" (UID: "f89ab83e-c555-4652-9cc4-9d43af3bb0c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:14:20.764910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.764890    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-kube-api-access-979ml" pod "f89ab83e-c555-4652-9cc4-9d43af3bb0c3" (UID: "f89ab83e-c555-4652-9cc4-9d43af3bb0c3"). InnerVolumeSpecName "kube-api-access-979ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:20.864812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.864744    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:20.864812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:20.864771    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-979ml\" (UniqueName: \"kubernetes.io/projected/f89ab83e-c555-4652-9cc4-9d43af3bb0c3-kube-api-access-979ml\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:21.067685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:21.067647    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 14:14:21.067893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:21.067703    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 14:14:21.129839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:21.127072    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 14:14:21.129839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:21.127134    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 14:14:21.158004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:21.157978    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"
Nov 02 14:14:21.158422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:21.158396    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv" event={"ID":"f89ab83e-c555-4652-9cc4-9d43af3bb0c3","Type":"ContainerDied","Data":"1d62bc918dc7adfbe5718a1c59f376a7ddffa0c4c8e94cb5db3e99ed95642b1c"}
Nov 02 14:14:21.158541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:21.158530    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d62bc918dc7adfbe5718a1c59f376a7ddffa0c4c8e94cb5db3e99ed95642b1c"
Nov 02 14:14:22.069866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:22.068137    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 14:14:22.069866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:22.068200    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 14:14:22.142798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:22.137304    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 14:14:22.142798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:22.137363    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 14:14:23.067674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:23.067641    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 14:14:23.068191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:23.067698    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 14:14:23.148987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:23.148954    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 14:14:23.149234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:23.149206    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 14:14:24.067651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:24.067616    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 02 14:14:24.068050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:24.068011    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 02 14:14:24.124758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:24.124715    5034 patch_prober.go:29] interesting pod/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Nov 02 14:14:24.124992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:24.124814    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Nov 02 14:14:25.222269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:25.222228    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"]
Nov 02 14:14:25.267092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:25.267055    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"]
Nov 02 14:14:25.509478 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:25.509446    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:25.511014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:25.510986    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:25.512132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:25.512092    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:14:25.565476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:25.565439    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:25.565855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:25.565765    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:25.566737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:25.566715    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:14:26.021877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:26.021837    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:14:26.023225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:26.023201    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:14:27.084286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.084246    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:14:27.892094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.892061    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"]
Nov 02 14:14:27.892848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.892831    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f89ab83e-c555-4652-9cc4-9d43af3bb0c3" containerName="revoke-keys"
Nov 02 14:14:27.892921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.892849    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89ab83e-c555-4652-9cc4-9d43af3bb0c3" containerName="revoke-keys"
Nov 02 14:14:27.892984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.892958    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89ab83e-c555-4652-9cc4-9d43af3bb0c3" containerName="revoke-keys"
Nov 02 14:14:27.897345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.897325    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.981982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.981947    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bd004e-2935-4b86-86ad-66d267777053-istio-podinfo\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.982199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.982000    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-certs\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.982199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.982118    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-envoy\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.982199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.982171    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-istio-token\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.982948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.982315    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-data\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.982948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.982373    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bd004e-2935-4b86-86ad-66d267777053-istiod-ca-cert\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.982948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.982411    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.982948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.982533    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m68sl\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-kube-api-access-m68sl\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.982948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.982698    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-credential-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:27.994848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:27.993796    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"]
Nov 02 14:14:28.020569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.020542    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-21baa7-openshift-default-dockercfg-wj446\""
Nov 02 14:14:28.084592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.084558    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m68sl\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-kube-api-access-m68sl\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.085099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.084693    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-credential-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.085099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.084863    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bd004e-2935-4b86-86ad-66d267777053-istio-podinfo\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.085099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.084899    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-certs\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.085099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.084954    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-envoy\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.085099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.084995    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-istio-token\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.085099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.085033    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-data\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.085099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.085056    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bd004e-2935-4b86-86ad-66d267777053-istiod-ca-cert\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.085099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.085086    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.085533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.085162    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-credential-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.086077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.086050    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-certs\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.086184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.086054    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-data\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.086247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.086180    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-socket\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.086750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.086724    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bd004e-2935-4b86-86ad-66d267777053-istiod-ca-cert\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.087829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.087762    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bd004e-2935-4b86-86ad-66d267777053-istio-podinfo\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.088291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.088259    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-envoy\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.124127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.124107    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:14:28.136479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.136449    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-istio-token\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.136580 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.136475    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m68sl\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-kube-api-access-m68sl\") pod \"e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") " pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.249809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.249067    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:28.713792 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:28.713747    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bd004e_2935_4b86_86ad_66d267777053.slice/crio-4501e3c8cf06e00be43209315bdccb875cda895f512437dec5d535b3f92dc873 WatchSource:0}: Error finding container 4501e3c8cf06e00be43209315bdccb875cda895f512437dec5d535b3f92dc873: Status 404 returned error can't find the container with id 4501e3c8cf06e00be43209315bdccb875cda895f512437dec5d535b3f92dc873
Nov 02 14:14:28.715603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.715578    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"]
Nov 02 14:14:28.716454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.716424    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:28.716520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.716488    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:28.716567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:28.716517    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:29.501577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:29.501530    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" event={"ID":"83bd004e-2935-4b86-86ad-66d267777053","Type":"ContainerStarted","Data":"81992a96fa8c1d7d59d976f6a107641bcea05e312a2ee54eed75833e62922330"}
Nov 02 14:14:29.502147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:29.501599    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" event={"ID":"83bd004e-2935-4b86-86ad-66d267777053","Type":"ContainerStarted","Data":"4501e3c8cf06e00be43209315bdccb875cda895f512437dec5d535b3f92dc873"}
Nov 02 14:14:29.580524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:29.580472    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" podStartSLOduration=2.580458923 podStartE2EDuration="2.580458923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:29.566521999 +0000 UTC m=+1860.317826438" watchObservedRunningTime="2026-11-02 14:14:29.580458923 +0000 UTC m=+1860.331763361"
Nov 02 14:14:30.227413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.227368    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct" podUID="d7836e47-b5cc-4583-9869-8555040ecd1f" containerName="payload-processing" containerID="cri-o://e9c7a9ccfa6959ff958c9bc783545233272c68d25f0bc5f5e2803df0b5b31ecf" gracePeriod=25
Nov 02 14:14:30.260862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.258383    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:30.273808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.273236    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:30.273808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.273590    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd" podUID="e7f497c9-0630-457f-99e9-5e125a849bfa" containerName="payload-pre-processing" containerID="cri-o://ce15ebe2c23b02ac3117e1a95e8e87549d02854b52682e5f0a73fb4503ba11e5" gracePeriod=25
Nov 02 14:14:30.763864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.755827    5034 generic.go:356] "Generic (PLEG): container finished" podID="d7836e47-b5cc-4583-9869-8555040ecd1f" containerID="e9c7a9ccfa6959ff958c9bc783545233272c68d25f0bc5f5e2803df0b5b31ecf" exitCode=0
Nov 02 14:14:30.763864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.757563    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct" event={"ID":"d7836e47-b5cc-4583-9869-8555040ecd1f","Type":"ContainerDied","Data":"e9c7a9ccfa6959ff958c9bc783545233272c68d25f0bc5f5e2803df0b5b31ecf"}
Nov 02 14:14:30.782387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.782353    5034 generic.go:356] "Generic (PLEG): container finished" podID="e7f497c9-0630-457f-99e9-5e125a849bfa" containerID="ce15ebe2c23b02ac3117e1a95e8e87549d02854b52682e5f0a73fb4503ba11e5" exitCode=0
Nov 02 14:14:30.784249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.784220    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd" event={"ID":"e7f497c9-0630-457f-99e9-5e125a849bfa","Type":"ContainerDied","Data":"ce15ebe2c23b02ac3117e1a95e8e87549d02854b52682e5f0a73fb4503ba11e5"}
Nov 02 14:14:30.786680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.786660    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:30.799999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.799979    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:14:30.813043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.811559    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:14:30.906804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.905934    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7836e47-b5cc-4583-9869-8555040ecd1f-kube-api-access-dqh9v\" (UniqueName: \"kubernetes.io/projected/d7836e47-b5cc-4583-9869-8555040ecd1f-kube-api-access-dqh9v\") pod \"d7836e47-b5cc-4583-9869-8555040ecd1f\" (UID: \"d7836e47-b5cc-4583-9869-8555040ecd1f\") "
Nov 02 14:14:30.906804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.906238    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7836e47-b5cc-4583-9869-8555040ecd1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7836e47-b5cc-4583-9869-8555040ecd1f-plugins-config-volume\") pod \"d7836e47-b5cc-4583-9869-8555040ecd1f\" (UID: \"d7836e47-b5cc-4583-9869-8555040ecd1f\") "
Nov 02 14:14:30.909435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.908213    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7836e47-b5cc-4583-9869-8555040ecd1f-plugins-config-volume" pod "d7836e47-b5cc-4583-9869-8555040ecd1f" (UID: "d7836e47-b5cc-4583-9869-8555040ecd1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:30.914371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:30.914344    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7836e47-b5cc-4583-9869-8555040ecd1f-kube-api-access-dqh9v" pod "d7836e47-b5cc-4583-9869-8555040ecd1f" (UID: "d7836e47-b5cc-4583-9869-8555040ecd1f"). InnerVolumeSpecName "kube-api-access-dqh9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:31.011858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.011756    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqh9v\" (UniqueName: \"kubernetes.io/projected/d7836e47-b5cc-4583-9869-8555040ecd1f-kube-api-access-dqh9v\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.011858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.011830    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7836e47-b5cc-4583-9869-8555040ecd1f-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.026720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.026313    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:14:31.216046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.216011    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f497c9-0630-457f-99e9-5e125a849bfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f497c9-0630-457f-99e9-5e125a849bfa-plugins-config-volume\") pod \"e7f497c9-0630-457f-99e9-5e125a849bfa\" (UID: \"e7f497c9-0630-457f-99e9-5e125a849bfa\") "
Nov 02 14:14:31.216291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.216271    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f497c9-0630-457f-99e9-5e125a849bfa-kube-api-access-9cprx\" (UniqueName: \"kubernetes.io/projected/e7f497c9-0630-457f-99e9-5e125a849bfa-kube-api-access-9cprx\") pod \"e7f497c9-0630-457f-99e9-5e125a849bfa\" (UID: \"e7f497c9-0630-457f-99e9-5e125a849bfa\") "
Nov 02 14:14:31.216434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.216412    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f497c9-0630-457f-99e9-5e125a849bfa-plugins-config-volume" pod "e7f497c9-0630-457f-99e9-5e125a849bfa" (UID: "e7f497c9-0630-457f-99e9-5e125a849bfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:31.217060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.217030    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f497c9-0630-457f-99e9-5e125a849bfa-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.218652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.218611    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f497c9-0630-457f-99e9-5e125a849bfa-kube-api-access-9cprx" pod "e7f497c9-0630-457f-99e9-5e125a849bfa" (UID: "e7f497c9-0630-457f-99e9-5e125a849bfa"). InnerVolumeSpecName "kube-api-access-9cprx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:31.330801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.330016    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cprx\" (UniqueName: \"kubernetes.io/projected/e7f497c9-0630-457f-99e9-5e125a849bfa-kube-api-access-9cprx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:31.828660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.827413    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct" event={"ID":"d7836e47-b5cc-4583-9869-8555040ecd1f","Type":"ContainerDied","Data":"ddac59c6a8e8e53d6ebe98417e218361e80e3a27589e8de8dcc3de0ea6ee1dc2"}
Nov 02 14:14:31.828660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.827490    5034 scope.go:122] "RemoveContainer" containerID="e9c7a9ccfa6959ff958c9bc783545233272c68d25f0bc5f5e2803df0b5b31ecf"
Nov 02 14:14:31.828660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.827666    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"
Nov 02 14:14:31.837352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.837329    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"
Nov 02 14:14:31.838048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.838023    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd" event={"ID":"e7f497c9-0630-457f-99e9-5e125a849bfa","Type":"ContainerDied","Data":"3d3bb1239cd939b2c183d7982c651760ac57bb5ffcf6524637711bb9adc8f520"}
Nov 02 14:14:31.900805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.893708    5034 scope.go:122] "RemoveContainer" containerID="ce15ebe2c23b02ac3117e1a95e8e87549d02854b52682e5f0a73fb4503ba11e5"
Nov 02 14:14:31.966857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.966827    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"]
Nov 02 14:14:31.981272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:31.981235    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-cdaf64-76465445cc-tnzct"]
Nov 02 14:14:32.024169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:32.024142    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"]
Nov 02 14:14:32.043800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:32.040807    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-cdaf64-58889fd65f-5qcpd"]
Nov 02 14:14:33.396219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:33.396182    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7836e47-b5cc-4583-9869-8555040ecd1f" path="/var/lib/kubelet/pods/d7836e47-b5cc-4583-9869-8555040ecd1f/volumes"
Nov 02 14:14:33.396832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:33.396810    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f497c9-0630-457f-99e9-5e125a849bfa" path="/var/lib/kubelet/pods/e7f497c9-0630-457f-99e9-5e125a849bfa/volumes"
Nov 02 14:14:34.140207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.140170    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"]
Nov 02 14:14:34.140874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.140856    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d7836e47-b5cc-4583-9869-8555040ecd1f" containerName="payload-processing"
Nov 02 14:14:34.140943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.140874    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7836e47-b5cc-4583-9869-8555040ecd1f" containerName="payload-processing"
Nov 02 14:14:34.140943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.140891    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e7f497c9-0630-457f-99e9-5e125a849bfa" containerName="payload-pre-processing"
Nov 02 14:14:34.140943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.140897    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f497c9-0630-457f-99e9-5e125a849bfa" containerName="payload-pre-processing"
Nov 02 14:14:34.141163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.141028    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f497c9-0630-457f-99e9-5e125a849bfa" containerName="payload-pre-processing"
Nov 02 14:14:34.141163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.141046    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7836e47-b5cc-4583-9869-8555040ecd1f" containerName="payload-processing"
Nov 02 14:14:34.145901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.145883    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:34.165044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.165025    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-21baa7-dockercfg-xp67z\""
Nov 02 14:14:34.165500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.165484    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-21baa7\""
Nov 02 14:14:34.180984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.180954    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"]
Nov 02 14:14:34.217813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.211026    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb4vd\" (UniqueName: \"kubernetes.io/projected/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-kube-api-access-xb4vd\") pod \"payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j\" (UID: \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:34.217813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.211146    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j\" (UID: \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:34.260812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.260104    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"]
Nov 02 14:14:34.276798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.276145    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:34.282334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.282298    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"]
Nov 02 14:14:34.323821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.323678    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb4vd\" (UniqueName: \"kubernetes.io/projected/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-kube-api-access-xb4vd\") pod \"payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j\" (UID: \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:34.323821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.323814    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j\" (UID: \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:34.324055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.323860    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwkzj\" (UniqueName: \"kubernetes.io/projected/17dd575a-b600-4a15-87e2-d211b76d7a40-kube-api-access-pwkzj\") pod \"payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm\" (UID: \"17dd575a-b600-4a15-87e2-d211b76d7a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:34.325269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.324151    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17dd575a-b600-4a15-87e2-d211b76d7a40-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm\" (UID: \"17dd575a-b600-4a15-87e2-d211b76d7a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:34.327913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.325507    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j\" (UID: \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:34.384319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.384288    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb4vd\" (UniqueName: \"kubernetes.io/projected/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-kube-api-access-xb4vd\") pod \"payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j\" (UID: \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:34.425875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.425751    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17dd575a-b600-4a15-87e2-d211b76d7a40-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm\" (UID: \"17dd575a-b600-4a15-87e2-d211b76d7a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:34.426269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.425922    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwkzj\" (UniqueName: \"kubernetes.io/projected/17dd575a-b600-4a15-87e2-d211b76d7a40-kube-api-access-pwkzj\") pod \"payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm\" (UID: \"17dd575a-b600-4a15-87e2-d211b76d7a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:34.426535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.426510    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17dd575a-b600-4a15-87e2-d211b76d7a40-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm\" (UID: \"17dd575a-b600-4a15-87e2-d211b76d7a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:34.453240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.453208    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwkzj\" (UniqueName: \"kubernetes.io/projected/17dd575a-b600-4a15-87e2-d211b76d7a40-kube-api-access-pwkzj\") pod \"payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm\" (UID: \"17dd575a-b600-4a15-87e2-d211b76d7a40\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:34.460039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.460009    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:34.639286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.639255    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:34.783701 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:34.783661    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e59015f_06e6_4f1b_90bb_58d210e1a0d5.slice/crio-d909c230ec32261eaef52e073889c878210a60fc4f30187c7f972a362791aad4 WatchSource:0}: Error finding container d909c230ec32261eaef52e073889c878210a60fc4f30187c7f972a362791aad4: Status 404 returned error can't find the container with id d909c230ec32261eaef52e073889c878210a60fc4f30187c7f972a362791aad4
Nov 02 14:14:34.786237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.786148    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"]
Nov 02 14:14:34.871502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.871469    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j" event={"ID":"6e59015f-06e6-4f1b-90bb-58d210e1a0d5","Type":"ContainerStarted","Data":"d909c230ec32261eaef52e073889c878210a60fc4f30187c7f972a362791aad4"}
Nov 02 14:14:34.872591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.872570    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:34.918222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.918158    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j" podStartSLOduration=0.918143458 podStartE2EDuration="918.143458ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:34.916442085 +0000 UTC m=+1865.667746523" watchObservedRunningTime="2026-11-02 14:14:34.918143458 +0000 UTC m=+1865.669447895"
Nov 02 14:14:34.954788 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:34.954423    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"]
Nov 02 14:14:35.879842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:35.879809    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm" event={"ID":"17dd575a-b600-4a15-87e2-d211b76d7a40","Type":"ContainerStarted","Data":"2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35"}
Nov 02 14:14:35.880200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:35.879866    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm" event={"ID":"17dd575a-b600-4a15-87e2-d211b76d7a40","Type":"ContainerStarted","Data":"18e26ffc1f315207d00cd1ca83206fa4ae0cfdb2500e9426ef57adbcc5e5aa66"}
Nov 02 14:14:35.880200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:35.879903    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:35.881613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:35.881592    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j" event={"ID":"6e59015f-06e6-4f1b-90bb-58d210e1a0d5","Type":"ContainerStarted","Data":"fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1"}
Nov 02 14:14:35.938633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:35.938578    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm" podStartSLOduration=1.938560372 podStartE2EDuration="1.938560372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:35.922153651 +0000 UTC m=+1866.673458102" watchObservedRunningTime="2026-11-02 14:14:35.938560372 +0000 UTC m=+1866.689864817"
Nov 02 14:14:40.130797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.126605    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"]
Nov 02 14:14:40.140828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.138102    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.143868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.141459    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"]
Nov 02 14:14:40.156194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.155729    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4c2811-openshift-default-dockercfg-pqrrz\""
Nov 02 14:14:40.313801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.252014    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.424885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.417879    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444082ae-6bfd-4126-8ae1-f30e98929081-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.424885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.417960    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444082ae-6bfd-4126-8ae1-f30e98929081-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.424885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.418004    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.424885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.418034    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.424885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.418081    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.424885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.418143    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.424885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.418469    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5tf7\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-kube-api-access-d5tf7\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.424885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.418528    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.523623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.523587    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444082ae-6bfd-4126-8ae1-f30e98929081-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.523856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.523639    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.523856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.523667    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.523856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.523709    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.523856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.523761    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.524084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.523866    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5tf7\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-kube-api-access-d5tf7\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.524084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.523909    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.524184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.524158    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.524305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.524192    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444082ae-6bfd-4126-8ae1-f30e98929081-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.524305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.524259    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444082ae-6bfd-4126-8ae1-f30e98929081-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.524838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.524756    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.524838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.524757    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.525057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.524909    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.525330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.525219    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.526651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.526623    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444082ae-6bfd-4126-8ae1-f30e98929081-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.527073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.527050    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.549946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.549921    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.550831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.550767    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5tf7\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-kube-api-access-d5tf7\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:40.759857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:40.759819    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:41.248802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:41.248097    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"]
Nov 02 14:14:41.260563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:41.256272    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:41.260563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:41.256375    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:41.260563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:41.256420    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:41.950846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:41.950407    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" event={"ID":"444082ae-6bfd-4126-8ae1-f30e98929081","Type":"ContainerStarted","Data":"c0e9bc5b929fd4d895707b2f47678c15e1cbb737b37e5cddce3d389d65cf59ac"}
Nov 02 14:14:41.950846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:41.950457    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" event={"ID":"444082ae-6bfd-4126-8ae1-f30e98929081","Type":"ContainerStarted","Data":"c81e560e3a9c6dec985d583e4678409c1143fe644b959101e36afaaa1e304890"}
Nov 02 14:14:42.011807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:42.010737    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" podStartSLOduration=2.010722008 podStartE2EDuration="2.010722008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:42.006118599 +0000 UTC m=+1872.757423038" watchObservedRunningTime="2026-11-02 14:14:42.010722008 +0000 UTC m=+1872.762026480"
Nov 02 14:14:42.760414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:42.760380    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:42.765652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:42.765627    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:42.958755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:42.958709    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:42.959750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:42.959725    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:14:43.537725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:43.537682    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"]
Nov 02 14:14:43.538101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:43.538047    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" containerID="cri-o://2f1a16efd621cf3ddc459928ecaffa01ca4802b2b47c7e5c44ff6643b77c698c" gracePeriod=30
Nov 02 14:14:43.543135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:43.543111    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"]
Nov 02 14:14:43.543300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:43.543272    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" containerID="cri-o://33549632a99f551df2578c2f84cc6aef7a495b50f0090a6af93e336910c4a25d" gracePeriod=30
Nov 02 14:14:44.224947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:44.224906    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:14:44.225363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:44.224983    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:46.894445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:46.894412    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:14:46.895003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:46.894485    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:14:48.060688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:48.060661    5034 generic.go:356] "Generic (PLEG): container finished" podID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerID="48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c" exitCode=0
Nov 02 14:14:48.061053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:48.060707    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" event={"ID":"5d46d575-e8a7-4171-8a7a-6ecd59b16728","Type":"ContainerDied","Data":"48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c"}
Nov 02 14:14:48.061155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:48.061139    5034 scope.go:122] "RemoveContainer" containerID="48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c"
Nov 02 14:14:48.063040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:48.062999    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:48.063135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:48.063100    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:48.063191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:48.063146    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:48.452889 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:14:48.452362    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:14:48.977951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:48.977867    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"]
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:48.988088    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:48.987983    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"]
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.014325    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-credential-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.014393    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhz5\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-kube-api-access-pjhz5\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.014474    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-envoy\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.014522    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-certs\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.014583    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-podinfo\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.014611    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-token\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.014676    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.014695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.014703    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istiod-ca-cert\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.015963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.014858    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-data\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.015963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.015430    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-2daae4-openshift-default-dockercfg-tf6d9\""
Nov 02 14:14:49.128801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.128372    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-credential-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.128801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.128457    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhz5\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-kube-api-access-pjhz5\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.128801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.128547    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-envoy\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.128801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.128594    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-certs\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.128801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.128659    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-podinfo\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.128801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.128687    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-token\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.128801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.128758    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.128801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.128799    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istiod-ca-cert\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.144571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.128899    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-data\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.144571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.129540    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-data\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.144571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.144110    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-credential-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.144571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.144411    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-socket\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.147054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.146715    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istiod-ca-cert\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.148084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.147848    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-certs\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.150151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.149232    5034 generic.go:356] "Generic (PLEG): container finished" podID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerID="2f1a16efd621cf3ddc459928ecaffa01ca4802b2b47c7e5c44ff6643b77c698c" exitCode=0
Nov 02 14:14:49.150151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.149341    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" event={"ID":"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4","Type":"ContainerDied","Data":"2f1a16efd621cf3ddc459928ecaffa01ca4802b2b47c7e5c44ff6643b77c698c"}
Nov 02 14:14:49.152708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.152437    5034 generic.go:356] "Generic (PLEG): container finished" podID="61d47968-77aa-4e90-8b68-d057588f5bef" containerID="33549632a99f551df2578c2f84cc6aef7a495b50f0090a6af93e336910c4a25d" exitCode=0
Nov 02 14:14:49.152708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.152594    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" event={"ID":"61d47968-77aa-4e90-8b68-d057588f5bef","Type":"ContainerDied","Data":"33549632a99f551df2578c2f84cc6aef7a495b50f0090a6af93e336910c4a25d"}
Nov 02 14:14:49.152708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.152640    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" event={"ID":"61d47968-77aa-4e90-8b68-d057588f5bef","Type":"ContainerDied","Data":"37bd3a4ac77939579ce1e5fb8e68de7e71e5ca0914a7d8142c6f20111143ea9f"}
Nov 02 14:14:49.152708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.152663    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37bd3a4ac77939579ce1e5fb8e68de7e71e5ca0914a7d8142c6f20111143ea9f"
Nov 02 14:14:49.154648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.154542    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-podinfo\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.157248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.157225    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" event={"ID":"5d46d575-e8a7-4171-8a7a-6ecd59b16728","Type":"ContainerStarted","Data":"cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3"}
Nov 02 14:14:49.159796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.159737    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-envoy\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.174398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.174201    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-token\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.234554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.233378    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhz5\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-kube-api-access-pjhz5\") pod \"e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") " pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.245217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.245182    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:49.251311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.251233    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:14:49.350772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.350722    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:14:49.441347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.441322    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-istio-token\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-istio-token\") pod \"61d47968-77aa-4e90-8b68-d057588f5bef\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") "
Nov 02 14:14:49.441616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.441595    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-kube-api-access-m4czm\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-kube-api-access-m4czm\") pod \"61d47968-77aa-4e90-8b68-d057588f5bef\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") "
Nov 02 14:14:49.441858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.441842    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d47968-77aa-4e90-8b68-d057588f5bef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d47968-77aa-4e90-8b68-d057588f5bef-istiod-ca-cert\") pod \"61d47968-77aa-4e90-8b68-d057588f5bef\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") "
Nov 02 14:14:49.442052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.442036    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61d47968-77aa-4e90-8b68-d057588f5bef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d47968-77aa-4e90-8b68-d057588f5bef-istio-podinfo\") pod \"61d47968-77aa-4e90-8b68-d057588f5bef\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") "
Nov 02 14:14:49.442584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.442564    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-credential-socket\") pod \"61d47968-77aa-4e90-8b68-d057588f5bef\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") "
Nov 02 14:14:49.442759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.442745    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-data\") pod \"61d47968-77aa-4e90-8b68-d057588f5bef\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") "
Nov 02 14:14:49.442955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.442943    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-certs\") pod \"61d47968-77aa-4e90-8b68-d057588f5bef\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") "
Nov 02 14:14:49.443054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.443045    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-socket\") pod \"61d47968-77aa-4e90-8b68-d057588f5bef\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") "
Nov 02 14:14:49.443244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.443229    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-envoy\") pod \"61d47968-77aa-4e90-8b68-d057588f5bef\" (UID: \"61d47968-77aa-4e90-8b68-d057588f5bef\") "
Nov 02 14:14:49.445606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.445583    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61d47968-77aa-4e90-8b68-d057588f5bef-istio-podinfo" pod "61d47968-77aa-4e90-8b68-d057588f5bef" (UID: "61d47968-77aa-4e90-8b68-d057588f5bef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:49.446502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.446480    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d47968-77aa-4e90-8b68-d057588f5bef-istiod-ca-cert" pod "61d47968-77aa-4e90-8b68-d057588f5bef" (UID: "61d47968-77aa-4e90-8b68-d057588f5bef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:49.446965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.446941    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-data" pod "61d47968-77aa-4e90-8b68-d057588f5bef" (UID: "61d47968-77aa-4e90-8b68-d057588f5bef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.447361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.447331    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-credential-socket" pod "61d47968-77aa-4e90-8b68-d057588f5bef" (UID: "61d47968-77aa-4e90-8b68-d057588f5bef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.447698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.447588    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-istio-token" pod "61d47968-77aa-4e90-8b68-d057588f5bef" (UID: "61d47968-77aa-4e90-8b68-d057588f5bef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.448011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.447986    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-certs" pod "61d47968-77aa-4e90-8b68-d057588f5bef" (UID: "61d47968-77aa-4e90-8b68-d057588f5bef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.448316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.448291    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-socket" pod "61d47968-77aa-4e90-8b68-d057588f5bef" (UID: "61d47968-77aa-4e90-8b68-d057588f5bef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.448911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.448878    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-envoy" pod "61d47968-77aa-4e90-8b68-d057588f5bef" (UID: "61d47968-77aa-4e90-8b68-d057588f5bef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.449598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.449572    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-kube-api-access-m4czm" pod "61d47968-77aa-4e90-8b68-d057588f5bef" (UID: "61d47968-77aa-4e90-8b68-d057588f5bef"). InnerVolumeSpecName "kube-api-access-m4czm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.545280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.545199    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-kube-api-access-2v748\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-kube-api-access-2v748\") pod \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") "
Nov 02 14:14:49.545580 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.545563    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-token\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-token\") pod \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") "
Nov 02 14:14:49.545702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.545688    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-credential-socket\") pod \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") "
Nov 02 14:14:49.546194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.545964    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-podinfo\") pod \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") "
Nov 02 14:14:49.546416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.546193    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-credential-socket" pod "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" (UID: "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.546527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.546376    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-socket\") pod \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") "
Nov 02 14:14:49.546652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.546638    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-data\") pod \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") "
Nov 02 14:14:49.547365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.547346    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-certs\") pod \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") "
Nov 02 14:14:49.547806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.547790    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-envoy\") pod \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") "
Nov 02 14:14:49.548375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.546899    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-data" pod "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" (UID: "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.548375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.548324    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istiod-ca-cert\") pod \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\" (UID: \"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4\") "
Nov 02 14:14:49.548375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.546942    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-socket" pod "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" (UID: "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.548375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.547658    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-certs" pod "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" (UID: "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.548836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.548771    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istiod-ca-cert" pod "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" (UID: "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:49.548944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.548922    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-token" pod "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" (UID: "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.549127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549103    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549130    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549147    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549160    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549173    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549185    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549198    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549210    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4czm\" (UniqueName: \"kubernetes.io/projected/61d47968-77aa-4e90-8b68-d057588f5bef-kube-api-access-m4czm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549223    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549235    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549248    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d47968-77aa-4e90-8b68-d057588f5bef-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549262    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d47968-77aa-4e90-8b68-d057588f5bef-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549274    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549286    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d47968-77aa-4e90-8b68-d057588f5bef-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549299    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.549663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.549277    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-kube-api-access-2v748" pod "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" (UID: "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4"). InnerVolumeSpecName "kube-api-access-2v748". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:49.550999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.550960    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-podinfo" pod "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" (UID: "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:49.552075 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.552050    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-envoy" pod "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" (UID: "c5486ccc-2e7b-427a-bc0b-8c6245dca2f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:49.607840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.607704    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:14:49.608341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.607968    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 14:14:49.608341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.608124    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 14:14:49.615154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.613818    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"]
Nov 02 14:14:49.615154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.614023    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy" containerID="cri-o://8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b" gracePeriod=30
Nov 02 14:14:49.638685 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:49.638663    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2a56dd_c87f_4e98_91c8_e628aac57f7b.slice/crio-69afc11fe8a6f1223f6207ddb0e7facf8e734b55b6d95639052781cd34327122 WatchSource:0}: Error finding container 69afc11fe8a6f1223f6207ddb0e7facf8e734b55b6d95639052781cd34327122: Status 404 returned error can't find the container with id 69afc11fe8a6f1223f6207ddb0e7facf8e734b55b6d95639052781cd34327122
Nov 02 14:14:49.641256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.641234    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"]
Nov 02 14:14:49.642385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.642342    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:49.642487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.642416    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:49.642487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.642457    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:14:49.650711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.650688    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.650829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.650713    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v748\" (UniqueName: \"kubernetes.io/projected/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-kube-api-access-2v748\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:49.650829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:49.650723    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:50.014802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.013857    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:14:50.014802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.013929    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:50.057823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.057472    5034 patch_prober.go:29] interesting pod/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:14:50.057823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.057537    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:14:50.267667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.267571    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" event={"ID":"3c2a56dd-c87f-4e98-91c8-e628aac57f7b","Type":"ContainerStarted","Data":"8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f"}
Nov 02 14:14:50.281979 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.281948    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" event={"ID":"3c2a56dd-c87f-4e98-91c8-e628aac57f7b","Type":"ContainerStarted","Data":"69afc11fe8a6f1223f6207ddb0e7facf8e734b55b6d95639052781cd34327122"}
Nov 02 14:14:50.306882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.306847    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z" event={"ID":"c5486ccc-2e7b-427a-bc0b-8c6245dca2f4","Type":"ContainerDied","Data":"d6b9c06f787ec0aaa374318cc9b1a337e26393cb17fa37b39982b39de843854c"}
Nov 02 14:14:50.307160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.307114    5034 scope.go:122] "RemoveContainer" containerID="2f1a16efd621cf3ddc459928ecaffa01ca4802b2b47c7e5c44ff6643b77c698c"
Nov 02 14:14:50.313885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.285960    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"
Nov 02 14:14:50.338456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.285997    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"
Nov 02 14:14:50.338456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.333318    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" podStartSLOduration=2.333300735 podStartE2EDuration="2.333300735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:50.328742898 +0000 UTC m=+1881.080047337" watchObservedRunningTime="2026-11-02 14:14:50.333300735 +0000 UTC m=+1881.084605240"
Nov 02 14:14:50.628855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.600076    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 14:14:50.629143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.629106    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 14:14:50.629801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.629753    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"]
Nov 02 14:14:50.648041 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:14:50.648001    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d47968_77aa_4e90_8b68_d057588f5bef.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5486ccc_2e7b_427a_bc0b_8c6245dca2f4.slice/crio-d6b9c06f787ec0aaa374318cc9b1a337e26393cb17fa37b39982b39de843854c\": RecentStats: unable to find data in memory cache]"
Nov 02 14:14:50.648538 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.648475    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-m5p9z"]
Nov 02 14:14:50.661566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.661528    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"]
Nov 02 14:14:50.666387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:50.666355    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-cdaf64-openshift-default-79d8b946b5-tpsmk"]
Nov 02 14:14:51.246043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:51.246005    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:51.251466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:51.251439    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:51.302584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:51.302557    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:51.303577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:51.303556    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:14:51.418170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:51.418027    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" path="/var/lib/kubelet/pods/61d47968-77aa-4e90-8b68-d057588f5bef/volumes"
Nov 02 14:14:51.419212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:51.419180    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" path="/var/lib/kubelet/pods/c5486ccc-2e7b-427a-bc0b-8c6245dca2f4/volumes"
Nov 02 14:14:51.606865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:51.597179    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 14:14:51.606865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:51.597239    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 14:14:52.596352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:52.595170    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 14:14:52.596352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:52.595230    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 14:14:53.594126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:53.594090    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 02 14:14:53.594316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:53.594144    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 02 14:14:54.134874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:54.134839    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6cbvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:14:54.135305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:54.134896    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:14:54.603679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:54.603646    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:14:54.604241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:54.604219    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:14:54.607357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:54.607335    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:14:54.940271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:54.940245    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 14:14:55.053668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.053630    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-envoy\") pod \"b5cc21d6-0861-4473-9761-e9072aa40b46\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") "
Nov 02 14:14:55.053899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.053881    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-socket\") pod \"b5cc21d6-0861-4473-9761-e9072aa40b46\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") "
Nov 02 14:14:55.053974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.053928    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5cc21d6-0861-4473-9761-e9072aa40b46-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc21d6-0861-4473-9761-e9072aa40b46-istiod-ca-cert\") pod \"b5cc21d6-0861-4473-9761-e9072aa40b46\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") "
Nov 02 14:14:55.053974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.053954    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-certs\") pod \"b5cc21d6-0861-4473-9761-e9072aa40b46\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") "
Nov 02 14:14:55.054096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054000    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-kube-api-access-xcvqd\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-kube-api-access-xcvqd\") pod \"b5cc21d6-0861-4473-9761-e9072aa40b46\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") "
Nov 02 14:14:55.054096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054058    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-podinfo\") pod \"b5cc21d6-0861-4473-9761-e9072aa40b46\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") "
Nov 02 14:14:55.054212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054151    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-data\") pod \"b5cc21d6-0861-4473-9761-e9072aa40b46\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") "
Nov 02 14:14:55.054212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054184    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-socket" pod "b5cc21d6-0861-4473-9761-e9072aa40b46" (UID: "b5cc21d6-0861-4473-9761-e9072aa40b46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.054313 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054215    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-credential-socket\") pod \"b5cc21d6-0861-4473-9761-e9072aa40b46\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") "
Nov 02 14:14:55.054372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054308    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-certs" pod "b5cc21d6-0861-4473-9761-e9072aa40b46" (UID: "b5cc21d6-0861-4473-9761-e9072aa40b46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.054372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054325    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-token\") pod \"b5cc21d6-0861-4473-9761-e9072aa40b46\" (UID: \"b5cc21d6-0861-4473-9761-e9072aa40b46\") "
Nov 02 14:14:55.054482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054406    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-data" pod "b5cc21d6-0861-4473-9761-e9072aa40b46" (UID: "b5cc21d6-0861-4473-9761-e9072aa40b46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.054564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054542    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-credential-socket" pod "b5cc21d6-0861-4473-9761-e9072aa40b46" (UID: "b5cc21d6-0861-4473-9761-e9072aa40b46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.055006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.054975    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5cc21d6-0861-4473-9761-e9072aa40b46-istiod-ca-cert" pod "b5cc21d6-0861-4473-9761-e9072aa40b46" (UID: "b5cc21d6-0861-4473-9761-e9072aa40b46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:14:55.055173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.055153    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:55.055245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.055180    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:55.055245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.055195    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:55.055245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.055210    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5cc21d6-0861-4473-9761-e9072aa40b46-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:55.055245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.055224    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:55.056492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.056464    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-envoy" pod "b5cc21d6-0861-4473-9761-e9072aa40b46" (UID: "b5cc21d6-0861-4473-9761-e9072aa40b46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:14:55.057069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.057047    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-podinfo" pod "b5cc21d6-0861-4473-9761-e9072aa40b46" (UID: "b5cc21d6-0861-4473-9761-e9072aa40b46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:14:55.057534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.057217    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-token" pod "b5cc21d6-0861-4473-9761-e9072aa40b46" (UID: "b5cc21d6-0861-4473-9761-e9072aa40b46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:55.057534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.057279    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-kube-api-access-xcvqd" pod "b5cc21d6-0861-4473-9761-e9072aa40b46" (UID: "b5cc21d6-0861-4473-9761-e9072aa40b46"). InnerVolumeSpecName "kube-api-access-xcvqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:14:55.156887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.156854    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcvqd\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-kube-api-access-xcvqd\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:55.156887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.156886    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:55.157452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.156899    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:55.157452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.156910    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5cc21d6-0861-4473-9761-e9072aa40b46-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:14:55.354143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.354105    5034 generic.go:356] "Generic (PLEG): container finished" podID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerID="8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b" exitCode=0
Nov 02 14:14:55.354323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.354155    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"
Nov 02 14:14:55.354323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.354197    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" event={"ID":"b5cc21d6-0861-4473-9761-e9072aa40b46","Type":"ContainerDied","Data":"8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b"}
Nov 02 14:14:55.354323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.354294    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw" event={"ID":"b5cc21d6-0861-4473-9761-e9072aa40b46","Type":"ContainerDied","Data":"fb73912bd496642f883c4c19771cf97f1ea26c43b60c142398359cc669a2e3e0"}
Nov 02 14:14:55.354489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.354323    5034 scope.go:122] "RemoveContainer" containerID="8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b"
Nov 02 14:14:55.366004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.365981    5034 scope.go:122] "RemoveContainer" containerID="8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b"
Nov 02 14:14:55.366291 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:14:55.366271    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b\": container with ID starting with 8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b not found: ID does not exist" containerID="8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b"
Nov 02 14:14:55.366348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.366300    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b"} err="failed to get container status \"8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b\": rpc error: code = NotFound desc = could not find container \"8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b\": container with ID starting with 8b2ad3a68d567829ba2142f2d4cb86c8fcbfc67d2a1ac7efe36c29f7dfab803b not found: ID does not exist"
Nov 02 14:14:55.416770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.416715    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"]
Nov 02 14:14:55.423840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:55.423814    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6cbvw"]
Nov 02 14:14:56.161739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.161708    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"]
Nov 02 14:14:56.162409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.162392    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy"
Nov 02 14:14:56.162457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.162410    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy"
Nov 02 14:14:56.162457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.162438    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy"
Nov 02 14:14:56.162457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.162442    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy"
Nov 02 14:14:56.162563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.162458    5034 cpu_manager.go:414] "containerMap: removing container" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy"
Nov 02 14:14:56.162563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.162465    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy"
Nov 02 14:14:56.162563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.162561    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" containerName="istio-proxy"
Nov 02 14:14:56.162659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.162570    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d47968-77aa-4e90-8b68-d057588f5bef" containerName="istio-proxy"
Nov 02 14:14:56.162659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.162583    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5486ccc-2e7b-427a-bc0b-8c6245dca2f4" containerName="istio-proxy"
Nov 02 14:14:56.167588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.167567    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:14:56.187195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.187170    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-fcefcc\""
Nov 02 14:14:56.187532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.187501    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-fcefcc-dockercfg-kxlkr\""
Nov 02 14:14:56.193628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.193602    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"]
Nov 02 14:14:56.311800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.306889    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m\" (UID: \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:14:56.311800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.306963    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2r8p\" (UniqueName: \"kubernetes.io/projected/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-kube-api-access-n2r8p\") pod \"payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m\" (UID: \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:14:56.381799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.362200    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"]
Nov 02 14:14:56.418762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.409087    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m\" (UID: \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:14:56.418762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.409209    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2r8p\" (UniqueName: \"kubernetes.io/projected/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-kube-api-access-n2r8p\") pod \"payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m\" (UID: \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:14:56.418762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.411037    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m\" (UID: \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:14:56.440617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.440591    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"]
Nov 02 14:14:56.440921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.440899    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:14:56.448760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.448708    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2r8p\" (UniqueName: \"kubernetes.io/projected/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-kube-api-access-n2r8p\") pod \"payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m\" (UID: \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:14:56.482006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.481977    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:14:56.614233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.613599    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfwrn\" (UniqueName: \"kubernetes.io/projected/ecd91416-7106-4f87-80d6-3d283e12c1a5-kube-api-access-xfwrn\") pod \"payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr\" (UID: \"ecd91416-7106-4f87-80d6-3d283e12c1a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:14:56.614233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.613701    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd91416-7106-4f87-80d6-3d283e12c1a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr\" (UID: \"ecd91416-7106-4f87-80d6-3d283e12c1a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:14:56.715051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.714967    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfwrn\" (UniqueName: \"kubernetes.io/projected/ecd91416-7106-4f87-80d6-3d283e12c1a5-kube-api-access-xfwrn\") pod \"payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr\" (UID: \"ecd91416-7106-4f87-80d6-3d283e12c1a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:14:56.715200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.715052    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd91416-7106-4f87-80d6-3d283e12c1a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr\" (UID: \"ecd91416-7106-4f87-80d6-3d283e12c1a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:14:56.715807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.715772    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd91416-7106-4f87-80d6-3d283e12c1a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr\" (UID: \"ecd91416-7106-4f87-80d6-3d283e12c1a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:14:56.739809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.739765    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfwrn\" (UniqueName: \"kubernetes.io/projected/ecd91416-7106-4f87-80d6-3d283e12c1a5-kube-api-access-xfwrn\") pod \"payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr\" (UID: \"ecd91416-7106-4f87-80d6-3d283e12c1a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:14:56.776634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.776605    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:14:56.823436 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:56.823404    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c3ad6b_ee98_4a76_86fe_ad4e9da28e0a.slice/crio-6a5cfe28f0bfb5ddf55ed0221ff7e5fdbe2b8b8fd4f545423c8c0e1582f3f001 WatchSource:0}: Error finding container 6a5cfe28f0bfb5ddf55ed0221ff7e5fdbe2b8b8fd4f545423c8c0e1582f3f001: Status 404 returned error can't find the container with id 6a5cfe28f0bfb5ddf55ed0221ff7e5fdbe2b8b8fd4f545423c8c0e1582f3f001
Nov 02 14:14:56.825029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:56.825002    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"]
Nov 02 14:14:57.102317 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:14:57.102281    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd91416_7106_4f87_80d6_3d283e12c1a5.slice/crio-b436a21c277fd59de50ff085a3afae02870a45779a48edb2be1f39fe6e6dd579 WatchSource:0}: Error finding container b436a21c277fd59de50ff085a3afae02870a45779a48edb2be1f39fe6e6dd579: Status 404 returned error can't find the container with id b436a21c277fd59de50ff085a3afae02870a45779a48edb2be1f39fe6e6dd579
Nov 02 14:14:57.106926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.106898    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"]
Nov 02 14:14:57.399295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.399151    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5cc21d6-0861-4473-9761-e9072aa40b46" path="/var/lib/kubelet/pods/b5cc21d6-0861-4473-9761-e9072aa40b46/volumes"
Nov 02 14:14:57.446191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.446156    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m" event={"ID":"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a","Type":"ContainerStarted","Data":"ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67"}
Nov 02 14:14:57.446374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.446244    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m" event={"ID":"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a","Type":"ContainerStarted","Data":"6a5cfe28f0bfb5ddf55ed0221ff7e5fdbe2b8b8fd4f545423c8c0e1582f3f001"}
Nov 02 14:14:57.446374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.446275    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:14:57.448079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.448052    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr" event={"ID":"ecd91416-7106-4f87-80d6-3d283e12c1a5","Type":"ContainerStarted","Data":"7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158"}
Nov 02 14:14:57.448201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.448101    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr" event={"ID":"ecd91416-7106-4f87-80d6-3d283e12c1a5","Type":"ContainerStarted","Data":"b436a21c277fd59de50ff085a3afae02870a45779a48edb2be1f39fe6e6dd579"}
Nov 02 14:14:57.448201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.448129    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:14:57.491843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.491746    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m" podStartSLOduration=1.4917316010000001 podStartE2EDuration="1.491731601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:57.489277724 +0000 UTC m=+1888.240582164" watchObservedRunningTime="2026-11-02 14:14:57.491731601 +0000 UTC m=+1888.243036039"
Nov 02 14:14:57.534368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:14:57.534324    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr" podStartSLOduration=1.534313775 podStartE2EDuration="1.534313775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:14:57.532254636 +0000 UTC m=+1888.283559075" watchObservedRunningTime="2026-11-02 14:14:57.534313775 +0000 UTC m=+1888.285618224"
Nov 02 14:15:00.157119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.157087    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"]
Nov 02 14:15:00.162878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.162852    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"]
Nov 02 14:15:00.163050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.162978    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:00.169060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.169033    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"]
Nov 02 14:15:00.169206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.169166    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:00.173827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.173805    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:00.185807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.185263    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"]
Nov 02 14:15:00.196024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.195985    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:00.198984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.197373    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:15:00.198984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.197456    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:15:00.198984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.197964    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:15:00.198984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.198621    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"]
Nov 02 14:15:00.207825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.205325    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"]
Nov 02 14:15:00.214857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.211039    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"]
Nov 02 14:15:00.215602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.215573    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.264424    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"]
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.266481    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bed0ec-be72-4057-a2c5-746c7a9b478f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6\" (UID: \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.266548    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fr4c\" (UniqueName: \"kubernetes.io/projected/c2bed0ec-be72-4057-a2c5-746c7a9b478f-kube-api-access-7fr4c\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6\" (UID: \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.266677    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893815-s6tzl\" (UID: \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.266745    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9\" (UID: \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.266811    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rd4l\" (UniqueName: \"kubernetes.io/projected/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-kube-api-access-5rd4l\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9\" (UID: \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.266845    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8lcb\" (UniqueName: \"kubernetes.io/projected/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-kube-api-access-m8lcb\") pod \"maas-api-key-cleanup-29893815-s6tzl\" (UID: \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.266893    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6pcj\" (UniqueName: \"kubernetes.io/projected/42575ae9-31d2-4b23-8ab4-e0a3449dc406-kube-api-access-m6pcj\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw\" (UID: \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.266964    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29g9\" (UniqueName: \"kubernetes.io/projected/5b167495-f48d-4e8a-b684-685b5ef4d615-kube-api-access-g29g9\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4\" (UID: \"5b167495-f48d-4e8a-b684-685b5ef4d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.267128    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42575ae9-31d2-4b23-8ab4-e0a3449dc406-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw\" (UID: \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:00.270417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.267155    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b167495-f48d-4e8a-b684-685b5ef4d615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4\" (UID: \"5b167495-f48d-4e8a-b684-685b5ef4d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:00.285325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.285005    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"]
Nov 02 14:15:00.296835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.296760    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"]
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.374852    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rd4l\" (UniqueName: \"kubernetes.io/projected/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-kube-api-access-5rd4l\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9\" (UID: \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.374917    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8lcb\" (UniqueName: \"kubernetes.io/projected/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-kube-api-access-m8lcb\") pod \"maas-api-key-cleanup-29893815-s6tzl\" (UID: \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.374970    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6pcj\" (UniqueName: \"kubernetes.io/projected/42575ae9-31d2-4b23-8ab4-e0a3449dc406-kube-api-access-m6pcj\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw\" (UID: \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.375049    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g29g9\" (UniqueName: \"kubernetes.io/projected/5b167495-f48d-4e8a-b684-685b5ef4d615-kube-api-access-g29g9\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4\" (UID: \"5b167495-f48d-4e8a-b684-685b5ef4d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.375188    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42575ae9-31d2-4b23-8ab4-e0a3449dc406-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw\" (UID: \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.375219    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b167495-f48d-4e8a-b684-685b5ef4d615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4\" (UID: \"5b167495-f48d-4e8a-b684-685b5ef4d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.375297    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bed0ec-be72-4057-a2c5-746c7a9b478f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6\" (UID: \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.375351    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fr4c\" (UniqueName: \"kubernetes.io/projected/c2bed0ec-be72-4057-a2c5-746c7a9b478f-kube-api-access-7fr4c\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6\" (UID: \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.375484    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893815-s6tzl\" (UID: \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.377083    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b167495-f48d-4e8a-b684-685b5ef4d615-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4\" (UID: \"5b167495-f48d-4e8a-b684-685b5ef4d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.377215    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42575ae9-31d2-4b23-8ab4-e0a3449dc406-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw\" (UID: \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.377710    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bed0ec-be72-4057-a2c5-746c7a9b478f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6\" (UID: \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.380041    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893815-s6tzl\" (UID: \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.380153    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9\" (UID: \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:00.384561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.381031    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9\" (UID: \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:00.392707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.392677    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"]
Nov 02 14:15:00.393451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.393396    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:00.415602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.397882    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"]
Nov 02 14:15:00.542215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.539604    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fr4c\" (UniqueName: \"kubernetes.io/projected/c2bed0ec-be72-4057-a2c5-746c7a9b478f-kube-api-access-7fr4c\") pod \"maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6\" (UID: \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:00.542215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.540198    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6pcj\" (UniqueName: \"kubernetes.io/projected/42575ae9-31d2-4b23-8ab4-e0a3449dc406-kube-api-access-m6pcj\") pod \"maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw\" (UID: \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:00.542215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.540799    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rd4l\" (UniqueName: \"kubernetes.io/projected/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-kube-api-access-5rd4l\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9\" (UID: \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:00.542215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.540901    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8lcb\" (UniqueName: \"kubernetes.io/projected/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-kube-api-access-m8lcb\") pod \"maas-api-key-cleanup-29893815-s6tzl\" (UID: \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:00.542215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.542041    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj\" (UID: \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:00.542215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.542149    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg7lr\" (UniqueName: \"kubernetes.io/projected/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-kube-api-access-jg7lr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj\" (UID: \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:00.544552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.543586    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29g9\" (UniqueName: \"kubernetes.io/projected/5b167495-f48d-4e8a-b684-685b5ef4d615-kube-api-access-g29g9\") pod \"maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4\" (UID: \"5b167495-f48d-4e8a-b684-685b5ef4d615\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:00.553372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.545011    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:00.553372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.545076    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:00.553372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.552296    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"]
Nov 02 14:15:00.553372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.552399    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:00.570368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.570308    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"]
Nov 02 14:15:00.574490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.574466    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:00.601541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.601237    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:00.645038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.645005    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6\" (UID: \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:00.646700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.645258    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj\" (UID: \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:00.646700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.645368    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg7lr\" (UniqueName: \"kubernetes.io/projected/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-kube-api-access-jg7lr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj\" (UID: \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:00.646700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.645461    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps8nq\" (UniqueName: \"kubernetes.io/projected/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-kube-api-access-ps8nq\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6\" (UID: \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:00.646700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.646504    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj\" (UID: \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:00.686260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.682463    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg7lr\" (UniqueName: \"kubernetes.io/projected/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-kube-api-access-jg7lr\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj\" (UID: \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:00.750939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.750106    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps8nq\" (UniqueName: \"kubernetes.io/projected/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-kube-api-access-ps8nq\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6\" (UID: \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:00.750939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.750203    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6\" (UID: \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:00.751819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.751800    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6\" (UID: \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:00.780418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.780379    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps8nq\" (UniqueName: \"kubernetes.io/projected/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-kube-api-access-ps8nq\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6\" (UID: \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:00.835401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.835368    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:00.871603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.871362    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:00.888799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:00.888537    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:01.000208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.000120    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"]
Nov 02 14:15:01.025699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.017598    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"]
Nov 02 14:15:01.028364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.026884    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"]
Nov 02 14:15:01.036319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.036229    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"]
Nov 02 14:15:01.214624 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:15:01.210250    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b167495_f48d_4e8a_b684_685b5ef4d615.slice/crio-e182f7214988ae34250377aca7c883f221a68ebdecd5950e5ed81fc050f64381 WatchSource:0}: Error finding container e182f7214988ae34250377aca7c883f221a68ebdecd5950e5ed81fc050f64381: Status 404 returned error can't find the container with id e182f7214988ae34250377aca7c883f221a68ebdecd5950e5ed81fc050f64381
Nov 02 14:15:01.214624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.212162    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"]
Nov 02 14:15:01.276341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.276310    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"]
Nov 02 14:15:01.325548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.325506    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"]
Nov 02 14:15:01.418834 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:15:01.418763    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d3f836_bac2_42a5_8c33_8251fc8b9bc8.slice/crio-3b2d516f4a3acb29e876d65f2a7fcae8f21dfb4d3866a9710a54c592e873ba29 WatchSource:0}: Error finding container 3b2d516f4a3acb29e876d65f2a7fcae8f21dfb4d3866a9710a54c592e873ba29: Status 404 returned error can't find the container with id 3b2d516f4a3acb29e876d65f2a7fcae8f21dfb4d3866a9710a54c592e873ba29
Nov 02 14:15:01.598485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.598449    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" event={"ID":"95d3f836-bac2-42a5-8c33-8251fc8b9bc8","Type":"ContainerStarted","Data":"3b2d516f4a3acb29e876d65f2a7fcae8f21dfb4d3866a9710a54c592e873ba29"}
Nov 02 14:15:01.600006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.599937    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" event={"ID":"541c34f6-f73d-4cb6-9c91-ebb4b973e85d","Type":"ContainerStarted","Data":"4585d4ea84b8a01ac8b7759a0780d4f1b4828c8994f66dddfe728a762c4e5983"}
Nov 02 14:15:01.601664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.601638    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" event={"ID":"5b167495-f48d-4e8a-b684-685b5ef4d615","Type":"ContainerStarted","Data":"e182f7214988ae34250377aca7c883f221a68ebdecd5950e5ed81fc050f64381"}
Nov 02 14:15:01.603612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.603589    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" event={"ID":"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e","Type":"ContainerStarted","Data":"5129d3a0d7a5c14143fb4466f69074ebdf7ce0b07a5179893f8ce9e922e44722"}
Nov 02 14:15:01.605070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.605034    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" event={"ID":"42575ae9-31d2-4b23-8ab4-e0a3449dc406","Type":"ContainerStarted","Data":"1af46068ccb1e9066023c30c6095e91acb1f9d993ffa92be122378872938069b"}
Nov 02 14:15:01.606764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.606740    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" event={"ID":"ce6bd3ef-9a24-42bd-8372-2683bf34f30d","Type":"ContainerStarted","Data":"3af76d0cd5234c4abc7a3eb7fd03dba0aed9cf82206e9051d3c23aec801e02bf"}
Nov 02 14:15:01.608111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.608090    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" event={"ID":"c2bed0ec-be72-4057-a2c5-746c7a9b478f","Type":"ContainerStarted","Data":"c49fe973d6e3a158e145eb8a88d9bbaadf682f87a3c2ba2252c4b717e6c71f20"}
Nov 02 14:15:01.650525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.650470    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" podStartSLOduration=1.650455659 podStartE2EDuration="1.650455659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:01.646185028 +0000 UTC m=+1892.397489466" watchObservedRunningTime="2026-11-02 14:15:01.650455659 +0000 UTC m=+1892.401760100"
Nov 02 14:15:01.691768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:01.691718    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" podStartSLOduration=1.6917036109999999 podStartE2EDuration="1.691703611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:01.688334187 +0000 UTC m=+1892.439638626" watchObservedRunningTime="2026-11-02 14:15:01.691703611 +0000 UTC m=+1892.443008056"
Nov 02 14:15:02.619097 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.619057    5034 generic.go:356] "Generic (PLEG): container finished" podID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerID="5592dcd2da391054a5989610661a876336b51bf5e38ca40041f89ac2d4da827b" exitCode=60
Nov 02 14:15:02.619536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.619135    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" event={"ID":"c2bed0ec-be72-4057-a2c5-746c7a9b478f","Type":"ContainerDied","Data":"5592dcd2da391054a5989610661a876336b51bf5e38ca40041f89ac2d4da827b"}
Nov 02 14:15:02.619536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.619528    5034 scope.go:122] "RemoveContainer" containerID="5592dcd2da391054a5989610661a876336b51bf5e38ca40041f89ac2d4da827b"
Nov 02 14:15:02.621249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.621222    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" event={"ID":"95d3f836-bac2-42a5-8c33-8251fc8b9bc8","Type":"ContainerStarted","Data":"fada57fac70ace7691176d232fc4862dda3721aeeda2dfe5dc1e05e3513d61e4"}
Nov 02 14:15:02.623431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.623407    5034 generic.go:356] "Generic (PLEG): container finished" podID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerID="bf723539c7cec2655cd50ff1640633fa0e184188a8a721946a29acf9f5cc1264" exitCode=60
Nov 02 14:15:02.623532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.623476    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" event={"ID":"541c34f6-f73d-4cb6-9c91-ebb4b973e85d","Type":"ContainerDied","Data":"bf723539c7cec2655cd50ff1640633fa0e184188a8a721946a29acf9f5cc1264"}
Nov 02 14:15:02.623881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.623768    5034 scope.go:122] "RemoveContainer" containerID="bf723539c7cec2655cd50ff1640633fa0e184188a8a721946a29acf9f5cc1264"
Nov 02 14:15:02.628306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.628282    5034 generic.go:356] "Generic (PLEG): container finished" podID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerID="5f36b8f986179c0a1d7bbb006f2a27ff6e31a5fdc36887969ce8543427826404" exitCode=60
Nov 02 14:15:02.628394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.628375    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" event={"ID":"5b167495-f48d-4e8a-b684-685b5ef4d615","Type":"ContainerDied","Data":"5f36b8f986179c0a1d7bbb006f2a27ff6e31a5fdc36887969ce8543427826404"}
Nov 02 14:15:02.628593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.628580    5034 scope.go:122] "RemoveContainer" containerID="5f36b8f986179c0a1d7bbb006f2a27ff6e31a5fdc36887969ce8543427826404"
Nov 02 14:15:02.632162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.630249    5034 generic.go:356] "Generic (PLEG): container finished" podID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerID="45f1a8dae1f7cd28fa2269549f1a2fc7d438481902bc836452764e6731fd1781" exitCode=60
Nov 02 14:15:02.632162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.630335    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" event={"ID":"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e","Type":"ContainerDied","Data":"45f1a8dae1f7cd28fa2269549f1a2fc7d438481902bc836452764e6731fd1781"}
Nov 02 14:15:02.632162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.630576    5034 scope.go:122] "RemoveContainer" containerID="45f1a8dae1f7cd28fa2269549f1a2fc7d438481902bc836452764e6731fd1781"
Nov 02 14:15:02.632503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.632371    5034 generic.go:356] "Generic (PLEG): container finished" podID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerID="0bad2d25212292722828ff19c57873ef27aa084a6e5ed68e4fbbb0a8c556a704" exitCode=60
Nov 02 14:15:02.632503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.632448    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" event={"ID":"42575ae9-31d2-4b23-8ab4-e0a3449dc406","Type":"ContainerDied","Data":"0bad2d25212292722828ff19c57873ef27aa084a6e5ed68e4fbbb0a8c556a704"}
Nov 02 14:15:02.632739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.632718    5034 scope.go:122] "RemoveContainer" containerID="0bad2d25212292722828ff19c57873ef27aa084a6e5ed68e4fbbb0a8c556a704"
Nov 02 14:15:02.635844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.635816    5034 generic.go:356] "Generic (PLEG): container finished" podID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerID="4e43796205df55f98f95a7f9ca79cd550627db154f5e9f56357b8511f438a317" exitCode=60
Nov 02 14:15:02.635977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.635961    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" event={"ID":"ce6bd3ef-9a24-42bd-8372-2683bf34f30d","Type":"ContainerDied","Data":"4e43796205df55f98f95a7f9ca79cd550627db154f5e9f56357b8511f438a317"}
Nov 02 14:15:02.636281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:02.636262    5034 scope.go:122] "RemoveContainer" containerID="4e43796205df55f98f95a7f9ca79cd550627db154f5e9f56357b8511f438a317"
Nov 02 14:15:03.019622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.019562    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" podStartSLOduration=3.019540573 podStartE2EDuration="3.019540573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:02.993122145 +0000 UTC m=+1893.744426586" watchObservedRunningTime="2026-11-02 14:15:03.019540573 +0000 UTC m=+1893.770845013"
Nov 02 14:15:03.664557 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.664462    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" event={"ID":"c2bed0ec-be72-4057-a2c5-746c7a9b478f","Type":"ContainerStarted","Data":"5fbb62ed52e894a3c790fb3f1c3821fa98fcfb46507f774d72c9871afc07fe51"}
Nov 02 14:15:03.668359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.668319    5034 generic.go:356] "Generic (PLEG): container finished" podID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerID="fada57fac70ace7691176d232fc4862dda3721aeeda2dfe5dc1e05e3513d61e4" exitCode=60
Nov 02 14:15:03.668525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.668408    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" event={"ID":"95d3f836-bac2-42a5-8c33-8251fc8b9bc8","Type":"ContainerDied","Data":"fada57fac70ace7691176d232fc4862dda3721aeeda2dfe5dc1e05e3513d61e4"}
Nov 02 14:15:03.672802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.669127    5034 scope.go:122] "RemoveContainer" containerID="fada57fac70ace7691176d232fc4862dda3721aeeda2dfe5dc1e05e3513d61e4"
Nov 02 14:15:03.701800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.672986    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" event={"ID":"541c34f6-f73d-4cb6-9c91-ebb4b973e85d","Type":"ContainerStarted","Data":"6b8ca1d060bb2b29995bfcbb65c2b45111cc88ca83820eb70d492e8c29cad678"}
Nov 02 14:15:03.702880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.702846    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" event={"ID":"5b167495-f48d-4e8a-b684-685b5ef4d615","Type":"ContainerStarted","Data":"e8d6ddebaa07b09c7b2ec5511cc60ee63a6640733476586999fa18988f868908"}
Nov 02 14:15:03.703067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.703046    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" event={"ID":"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e","Type":"ContainerStarted","Data":"58f9f53d2767ee5cbf9a74d92e520ed02c41734fa161f105d65b8d2fd8f5b9a8"}
Nov 02 14:15:03.703202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.703188    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" event={"ID":"42575ae9-31d2-4b23-8ab4-e0a3449dc406","Type":"ContainerStarted","Data":"b44a8d9d8a5b7abd3ac2d94ab412827383ecce876f9033ca1bd7925ccdce6aa5"}
Nov 02 14:15:03.705330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.704324    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" event={"ID":"ce6bd3ef-9a24-42bd-8372-2683bf34f30d","Type":"ContainerStarted","Data":"fa82bfd231c303fe7ac099bbe25a8ce6f8a68fdf007caf12bae16f59ce3dc0ba"}
Nov 02 14:15:03.721803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.720719    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" podStartSLOduration=3.720703974 podStartE2EDuration="3.720703974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.720299065 +0000 UTC m=+1894.471603504" watchObservedRunningTime="2026-11-02 14:15:03.720703974 +0000 UTC m=+1894.472008413"
Nov 02 14:15:03.813761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.813692    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" podStartSLOduration=3.813672798 podStartE2EDuration="3.813672798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.768043374 +0000 UTC m=+1894.519347812" watchObservedRunningTime="2026-11-02 14:15:03.813672798 +0000 UTC m=+1894.564977255"
Nov 02 14:15:03.897529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:03.892398    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" podStartSLOduration=3.892375652 podStartE2EDuration="3.892375652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:03.889396148 +0000 UTC m=+1894.640700589" watchObservedRunningTime="2026-11-02 14:15:03.892375652 +0000 UTC m=+1894.643680092"
Nov 02 14:15:04.065458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.065394    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" podStartSLOduration=4.065371179 podStartE2EDuration="4.065371179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:04.062250231 +0000 UTC m=+1894.813554671" watchObservedRunningTime="2026-11-02 14:15:04.065371179 +0000 UTC m=+1894.816675617"
Nov 02 14:15:04.730028 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.729988    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"]
Nov 02 14:15:04.736729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.736234    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:04.741118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.740898    5034 generic.go:356] "Generic (PLEG): container finished" podID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerID="5fbb62ed52e894a3c790fb3f1c3821fa98fcfb46507f774d72c9871afc07fe51" exitCode=60
Nov 02 14:15:04.741118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.741000    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" event={"ID":"c2bed0ec-be72-4057-a2c5-746c7a9b478f","Type":"ContainerDied","Data":"5fbb62ed52e894a3c790fb3f1c3821fa98fcfb46507f774d72c9871afc07fe51"}
Nov 02 14:15:04.741118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.741069    5034 scope.go:122] "RemoveContainer" containerID="5592dcd2da391054a5989610661a876336b51bf5e38ca40041f89ac2d4da827b"
Nov 02 14:15:04.741355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.741295    5034 scope.go:122] "RemoveContainer" containerID="5fbb62ed52e894a3c790fb3f1c3821fa98fcfb46507f774d72c9871afc07fe51"
Nov 02 14:15:04.741717 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:04.741619    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6_odh-ai-gateway-infra(c2bed0ec-be72-4057-a2c5-746c7a9b478f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f"
Nov 02 14:15:04.748797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.747288    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd\" (UID: \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:04.748797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.747711    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nnjc\" (UniqueName: \"kubernetes.io/projected/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-kube-api-access-6nnjc\") pod \"payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd\" (UID: \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:04.752464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.751020    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" event={"ID":"95d3f836-bac2-42a5-8c33-8251fc8b9bc8","Type":"ContainerStarted","Data":"1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857"}
Nov 02 14:15:04.753065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.753037    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"]
Nov 02 14:15:04.755637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.755613    5034 generic.go:356] "Generic (PLEG): container finished" podID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerID="6b8ca1d060bb2b29995bfcbb65c2b45111cc88ca83820eb70d492e8c29cad678" exitCode=60
Nov 02 14:15:04.755744 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.755692    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" event={"ID":"541c34f6-f73d-4cb6-9c91-ebb4b973e85d","Type":"ContainerDied","Data":"6b8ca1d060bb2b29995bfcbb65c2b45111cc88ca83820eb70d492e8c29cad678"}
Nov 02 14:15:04.756019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.755996    5034 scope.go:122] "RemoveContainer" containerID="6b8ca1d060bb2b29995bfcbb65c2b45111cc88ca83820eb70d492e8c29cad678"
Nov 02 14:15:04.756389 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:04.756354    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj_odh-ai-gateway-infra(541c34f6-f73d-4cb6-9c91-ebb4b973e85d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d"
Nov 02 14:15:04.756987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.756967    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-2daae4-dockercfg-9422t\""
Nov 02 14:15:04.757996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.757971    5034 generic.go:356] "Generic (PLEG): container finished" podID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerID="e8d6ddebaa07b09c7b2ec5511cc60ee63a6640733476586999fa18988f868908" exitCode=60
Nov 02 14:15:04.758073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.758047    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" event={"ID":"5b167495-f48d-4e8a-b684-685b5ef4d615","Type":"ContainerDied","Data":"e8d6ddebaa07b09c7b2ec5511cc60ee63a6640733476586999fa18988f868908"}
Nov 02 14:15:04.758289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.758272    5034 scope.go:122] "RemoveContainer" containerID="e8d6ddebaa07b09c7b2ec5511cc60ee63a6640733476586999fa18988f868908"
Nov 02 14:15:04.758583 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:04.758559    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4_odh-ai-gateway-infra(5b167495-f48d-4e8a-b684-685b5ef4d615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615"
Nov 02 14:15:04.760158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.760137    5034 generic.go:356] "Generic (PLEG): container finished" podID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerID="58f9f53d2767ee5cbf9a74d92e520ed02c41734fa161f105d65b8d2fd8f5b9a8" exitCode=60
Nov 02 14:15:04.760241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.760216    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" event={"ID":"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e","Type":"ContainerDied","Data":"58f9f53d2767ee5cbf9a74d92e520ed02c41734fa161f105d65b8d2fd8f5b9a8"}
Nov 02 14:15:04.760442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.760426    5034 scope.go:122] "RemoveContainer" containerID="58f9f53d2767ee5cbf9a74d92e520ed02c41734fa161f105d65b8d2fd8f5b9a8"
Nov 02 14:15:04.760727 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:04.760703    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893815-s6tzl_odh-ai-gateway-infra(ed7a63da-d98b-4cc7-b666-4ce4f2eee86e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e"
Nov 02 14:15:04.762281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.762259    5034 generic.go:356] "Generic (PLEG): container finished" podID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerID="b44a8d9d8a5b7abd3ac2d94ab412827383ecce876f9033ca1bd7925ccdce6aa5" exitCode=60
Nov 02 14:15:04.762382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.762351    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" event={"ID":"42575ae9-31d2-4b23-8ab4-e0a3449dc406","Type":"ContainerDied","Data":"b44a8d9d8a5b7abd3ac2d94ab412827383ecce876f9033ca1bd7925ccdce6aa5"}
Nov 02 14:15:04.763372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.762537    5034 scope.go:122] "RemoveContainer" containerID="b44a8d9d8a5b7abd3ac2d94ab412827383ecce876f9033ca1bd7925ccdce6aa5"
Nov 02 14:15:04.763372 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:04.762817    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw_odh-ai-gateway-infra(42575ae9-31d2-4b23-8ab4-e0a3449dc406)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406"
Nov 02 14:15:04.763372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.762945    5034 scope.go:122] "RemoveContainer" containerID="bf723539c7cec2655cd50ff1640633fa0e184188a8a721946a29acf9f5cc1264"
Nov 02 14:15:04.765043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.765017    5034 generic.go:356] "Generic (PLEG): container finished" podID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerID="fa82bfd231c303fe7ac099bbe25a8ce6f8a68fdf007caf12bae16f59ce3dc0ba" exitCode=60
Nov 02 14:15:04.765149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.765103    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" event={"ID":"ce6bd3ef-9a24-42bd-8372-2683bf34f30d","Type":"ContainerDied","Data":"fa82bfd231c303fe7ac099bbe25a8ce6f8a68fdf007caf12bae16f59ce3dc0ba"}
Nov 02 14:15:04.765347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.765329    5034 scope.go:122] "RemoveContainer" containerID="fa82bfd231c303fe7ac099bbe25a8ce6f8a68fdf007caf12bae16f59ce3dc0ba"
Nov 02 14:15:04.765637 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:04.765614    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9_odh-ai-gateway-infra(ce6bd3ef-9a24-42bd-8372-2683bf34f30d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d"
Nov 02 14:15:04.848809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.832941    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-2daae4\""
Nov 02 14:15:04.881831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.878096    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd\" (UID: \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:04.881831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.878438    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nnjc\" (UniqueName: \"kubernetes.io/projected/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-kube-api-access-6nnjc\") pod \"payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd\" (UID: \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:04.881831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:04.880675    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd\" (UID: \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:05.057262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.057151    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nnjc\" (UniqueName: \"kubernetes.io/projected/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-kube-api-access-6nnjc\") pod \"payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd\" (UID: \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:05.099181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.097016    5034 scope.go:122] "RemoveContainer" containerID="5f36b8f986179c0a1d7bbb006f2a27ff6e31a5fdc36887969ce8543427826404"
Nov 02 14:15:05.216951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.216923    5034 scope.go:122] "RemoveContainer" containerID="45f1a8dae1f7cd28fa2269549f1a2fc7d438481902bc836452764e6731fd1781"
Nov 02 14:15:05.232236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.232211    5034 scope.go:122] "RemoveContainer" containerID="0bad2d25212292722828ff19c57873ef27aa084a6e5ed68e4fbbb0a8c556a704"
Nov 02 14:15:05.250284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.250254    5034 scope.go:122] "RemoveContainer" containerID="4e43796205df55f98f95a7f9ca79cd550627db154f5e9f56357b8511f438a317"
Nov 02 14:15:05.355834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.355735    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:05.717749 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:15:05.717714    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e620544_fbdd_41c4_baf3_7bfc9aaf13ba.slice/crio-f0d9d1ba1eb2432a74ddafd4e8ceb9a862bec18066c147ab277cadd7b87c70f4 WatchSource:0}: Error finding container f0d9d1ba1eb2432a74ddafd4e8ceb9a862bec18066c147ab277cadd7b87c70f4: Status 404 returned error can't find the container with id f0d9d1ba1eb2432a74ddafd4e8ceb9a862bec18066c147ab277cadd7b87c70f4
Nov 02 14:15:05.722285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.722254    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"]
Nov 02 14:15:05.824858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.824822    5034 generic.go:356] "Generic (PLEG): container finished" podID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerID="1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857" exitCode=60
Nov 02 14:15:05.825228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.824897    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" event={"ID":"95d3f836-bac2-42a5-8c33-8251fc8b9bc8","Type":"ContainerDied","Data":"1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857"}
Nov 02 14:15:05.825228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.824968    5034 scope.go:122] "RemoveContainer" containerID="fada57fac70ace7691176d232fc4862dda3721aeeda2dfe5dc1e05e3513d61e4"
Nov 02 14:15:05.825366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.825352    5034 scope.go:122] "RemoveContainer" containerID="1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857"
Nov 02 14:15:05.825953 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:05.825930    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6_odh-ai-gateway-infra(95d3f836-bac2-42a5-8c33-8251fc8b9bc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8"
Nov 02 14:15:05.829689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.829667    5034 scope.go:122] "RemoveContainer" containerID="6b8ca1d060bb2b29995bfcbb65c2b45111cc88ca83820eb70d492e8c29cad678"
Nov 02 14:15:05.830132 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:05.830106    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj_odh-ai-gateway-infra(541c34f6-f73d-4cb6-9c91-ebb4b973e85d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d"
Nov 02 14:15:05.831948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.831929    5034 scope.go:122] "RemoveContainer" containerID="e8d6ddebaa07b09c7b2ec5511cc60ee63a6640733476586999fa18988f868908"
Nov 02 14:15:05.832342 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:05.832319    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4_odh-ai-gateway-infra(5b167495-f48d-4e8a-b684-685b5ef4d615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615"
Nov 02 14:15:05.834058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.834033    5034 scope.go:122] "RemoveContainer" containerID="58f9f53d2767ee5cbf9a74d92e520ed02c41734fa161f105d65b8d2fd8f5b9a8"
Nov 02 14:15:05.834427 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:05.834367    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893815-s6tzl_odh-ai-gateway-infra(ed7a63da-d98b-4cc7-b666-4ce4f2eee86e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e"
Nov 02 14:15:05.835875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.835812    5034 scope.go:122] "RemoveContainer" containerID="b44a8d9d8a5b7abd3ac2d94ab412827383ecce876f9033ca1bd7925ccdce6aa5"
Nov 02 14:15:05.836449 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:05.836131    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw_odh-ai-gateway-infra(42575ae9-31d2-4b23-8ab4-e0a3449dc406)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406"
Nov 02 14:15:05.837488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.837470    5034 scope.go:122] "RemoveContainer" containerID="fa82bfd231c303fe7ac099bbe25a8ce6f8a68fdf007caf12bae16f59ce3dc0ba"
Nov 02 14:15:05.837720 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:05.837702    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9_odh-ai-gateway-infra(ce6bd3ef-9a24-42bd-8372-2683bf34f30d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d"
Nov 02 14:15:05.839552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.839526    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd" event={"ID":"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba","Type":"ContainerStarted","Data":"f0d9d1ba1eb2432a74ddafd4e8ceb9a862bec18066c147ab277cadd7b87c70f4"}
Nov 02 14:15:05.843312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:05.843296    5034 scope.go:122] "RemoveContainer" containerID="5fbb62ed52e894a3c790fb3f1c3821fa98fcfb46507f774d72c9871afc07fe51"
Nov 02 14:15:05.843574 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:05.843556    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6_odh-ai-gateway-infra(c2bed0ec-be72-4057-a2c5-746c7a9b478f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f"
Nov 02 14:15:06.855149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:06.855102    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd" event={"ID":"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba","Type":"ContainerStarted","Data":"9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc"}
Nov 02 14:15:06.855570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:06.855201    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:06.857262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:06.857242    5034 scope.go:122] "RemoveContainer" containerID="1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857"
Nov 02 14:15:06.857550 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:06.857531    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6_odh-ai-gateway-infra(95d3f836-bac2-42a5-8c33-8251fc8b9bc8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8"
Nov 02 14:15:06.909872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:06.909805    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd" podStartSLOduration=2.909760901 podStartE2EDuration="2.909760901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:06.906679074 +0000 UTC m=+1897.657983511" watchObservedRunningTime="2026-11-02 14:15:06.909760901 +0000 UTC m=+1897.661065369"
Nov 02 14:15:08.459195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:08.459110    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:15:08.459195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:08.459191    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:15:12.935958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:12.935923    5034 generic.go:356] "Generic (PLEG): container finished" podID="83bd004e-2935-4b86-86ad-66d267777053" containerID="81992a96fa8c1d7d59d976f6a107641bcea05e312a2ee54eed75833e62922330" exitCode=0
Nov 02 14:15:12.936529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:12.935995    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" event={"ID":"83bd004e-2935-4b86-86ad-66d267777053","Type":"ContainerDied","Data":"81992a96fa8c1d7d59d976f6a107641bcea05e312a2ee54eed75833e62922330"}
Nov 02 14:15:12.936529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:12.936485    5034 scope.go:122] "RemoveContainer" containerID="81992a96fa8c1d7d59d976f6a107641bcea05e312a2ee54eed75833e62922330"
Nov 02 14:15:12.938273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:12.938242    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:12.938360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:12.938313    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:12.938360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:12.938347    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:13.949699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:13.949661    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" event={"ID":"83bd004e-2935-4b86-86ad-66d267777053","Type":"ContainerStarted","Data":"7be0d0d64e49f0e486a906badced715b1414f1f83d074d7eade1bdc41505c7fc"}
Nov 02 14:15:14.258115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:14.258075    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:15:14.259532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:14.259504    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 02 14:15:14.259682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:14.259559    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 02 14:15:15.258120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:15.258084    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 02 14:15:15.258542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:15.258139    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 02 14:15:15.395513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:15.395479    5034 scope.go:122] "RemoveContainer" containerID="6b8ca1d060bb2b29995bfcbb65c2b45111cc88ca83820eb70d492e8c29cad678"
Nov 02 14:15:15.396260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:15.396234    5034 scope.go:122] "RemoveContainer" containerID="5fbb62ed52e894a3c790fb3f1c3821fa98fcfb46507f774d72c9871afc07fe51"
Nov 02 14:15:15.397484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:15.397460    5034 scope.go:122] "RemoveContainer" containerID="e8d6ddebaa07b09c7b2ec5511cc60ee63a6640733476586999fa18988f868908"
Nov 02 14:15:15.787482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:15.787399    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:15:16.258269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:16.258223    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 02 14:15:16.258752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:16.258298    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 02 14:15:16.389868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:16.389765    5034 scope.go:122] "RemoveContainer" containerID="b44a8d9d8a5b7abd3ac2d94ab412827383ecce876f9033ca1bd7925ccdce6aa5"
Nov 02 14:15:17.004003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.003971    5034 generic.go:356] "Generic (PLEG): container finished" podID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerID="008eec60ec5edcf29ec2bc40e51639029675fdd3044743b1152faec55e0ed1db" exitCode=60
Nov 02 14:15:17.004121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.004040    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" event={"ID":"c2bed0ec-be72-4057-a2c5-746c7a9b478f","Type":"ContainerDied","Data":"008eec60ec5edcf29ec2bc40e51639029675fdd3044743b1152faec55e0ed1db"}
Nov 02 14:15:17.004178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.004130    5034 scope.go:122] "RemoveContainer" containerID="5fbb62ed52e894a3c790fb3f1c3821fa98fcfb46507f774d72c9871afc07fe51"
Nov 02 14:15:17.004870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.004624    5034 scope.go:122] "RemoveContainer" containerID="008eec60ec5edcf29ec2bc40e51639029675fdd3044743b1152faec55e0ed1db"
Nov 02 14:15:17.005579 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:17.005549    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6_odh-ai-gateway-infra(c2bed0ec-be72-4057-a2c5-746c7a9b478f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f"
Nov 02 14:15:17.007505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.007480    5034 generic.go:356] "Generic (PLEG): container finished" podID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerID="f50d3d17b633f78de8d2a31dd88eef4950b42c77bea14a547c81b07d6289c8b7" exitCode=60
Nov 02 14:15:17.007628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.007515    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" event={"ID":"541c34f6-f73d-4cb6-9c91-ebb4b973e85d","Type":"ContainerDied","Data":"f50d3d17b633f78de8d2a31dd88eef4950b42c77bea14a547c81b07d6289c8b7"}
Nov 02 14:15:17.008054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.008023    5034 scope.go:122] "RemoveContainer" containerID="f50d3d17b633f78de8d2a31dd88eef4950b42c77bea14a547c81b07d6289c8b7"
Nov 02 14:15:17.008463 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:17.008430    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj_odh-ai-gateway-infra(541c34f6-f73d-4cb6-9c91-ebb4b973e85d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d"
Nov 02 14:15:17.010116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.010095    5034 generic.go:356] "Generic (PLEG): container finished" podID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerID="9a4f36961158ff19a6d30b42c03d5b38245514e358c083d309d8641c6f8424cc" exitCode=60
Nov 02 14:15:17.010212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.010160    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" event={"ID":"5b167495-f48d-4e8a-b684-685b5ef4d615","Type":"ContainerDied","Data":"9a4f36961158ff19a6d30b42c03d5b38245514e358c083d309d8641c6f8424cc"}
Nov 02 14:15:17.010541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.010526    5034 scope.go:122] "RemoveContainer" containerID="9a4f36961158ff19a6d30b42c03d5b38245514e358c083d309d8641c6f8424cc"
Nov 02 14:15:17.010866 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:17.010848    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4_odh-ai-gateway-infra(5b167495-f48d-4e8a-b684-685b5ef4d615)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615"
Nov 02 14:15:17.021038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.021019    5034 scope.go:122] "RemoveContainer" containerID="6b8ca1d060bb2b29995bfcbb65c2b45111cc88ca83820eb70d492e8c29cad678"
Nov 02 14:15:17.034635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.034606    5034 scope.go:122] "RemoveContainer" containerID="e8d6ddebaa07b09c7b2ec5511cc60ee63a6640733476586999fa18988f868908"
Nov 02 14:15:17.258101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.258020    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 02 14:15:17.258101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.258074    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 02 14:15:17.405634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.405595    5034 scope.go:122] "RemoveContainer" containerID="1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857"
Nov 02 14:15:17.406144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.405761    5034 scope.go:122] "RemoveContainer" containerID="58f9f53d2767ee5cbf9a74d92e520ed02c41734fa161f105d65b8d2fd8f5b9a8"
Nov 02 14:15:17.895865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:17.895072    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:15:18.025122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.025081    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" event={"ID":"95d3f836-bac2-42a5-8c33-8251fc8b9bc8","Type":"ContainerStarted","Data":"6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0"}
Nov 02 14:15:18.037843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.037804    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" event={"ID":"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e","Type":"ContainerStarted","Data":"1b3a3dfcd577535a33cbb156a55d8a5531585924ab5653c4d909b2d4c4f26d93"}
Nov 02 14:15:18.041935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.041895    5034 generic.go:356] "Generic (PLEG): container finished" podID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerID="61316f92108b2432c8cdb9cf2afc555b8b60f2d6c22256b1a144698dae14cdc1" exitCode=60
Nov 02 14:15:18.042721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.042309    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" event={"ID":"42575ae9-31d2-4b23-8ab4-e0a3449dc406","Type":"ContainerDied","Data":"61316f92108b2432c8cdb9cf2afc555b8b60f2d6c22256b1a144698dae14cdc1"}
Nov 02 14:15:18.042721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.042397    5034 scope.go:122] "RemoveContainer" containerID="b44a8d9d8a5b7abd3ac2d94ab412827383ecce876f9033ca1bd7925ccdce6aa5"
Nov 02 14:15:18.043022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.043001    5034 scope.go:122] "RemoveContainer" containerID="61316f92108b2432c8cdb9cf2afc555b8b60f2d6c22256b1a144698dae14cdc1"
Nov 02 14:15:18.043658 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:18.043626    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw_odh-ai-gateway-infra(42575ae9-31d2-4b23-8ab4-e0a3449dc406)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406"
Nov 02 14:15:18.132959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.132918    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"]
Nov 02 14:15:18.149356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.149315    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"]
Nov 02 14:15:18.176814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.175848    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"]
Nov 02 14:15:18.276894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.261266    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body=
Nov 02 14:15:18.276894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.261332    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused"
Nov 02 14:15:18.425888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.408480    5034 scope.go:122] "RemoveContainer" containerID="fa82bfd231c303fe7ac099bbe25a8ce6f8a68fdf007caf12bae16f59ce3dc0ba"
Nov 02 14:15:18.519144 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:18.519106    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:15:18.610996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.610962    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:18.643689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.643659    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:18.717948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.716163    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-kube-api-access-jg7lr\" (UniqueName: \"kubernetes.io/projected/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-kube-api-access-jg7lr\") pod \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\" (UID: \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\") "
Nov 02 14:15:18.717948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.716302    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2bed0ec-be72-4057-a2c5-746c7a9b478f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bed0ec-be72-4057-a2c5-746c7a9b478f-maas-api-service-ca\") pod \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\" (UID: \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\") "
Nov 02 14:15:18.717948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.716345    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2bed0ec-be72-4057-a2c5-746c7a9b478f-kube-api-access-7fr4c\" (UniqueName: \"kubernetes.io/projected/c2bed0ec-be72-4057-a2c5-746c7a9b478f-kube-api-access-7fr4c\") pod \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\" (UID: \"c2bed0ec-be72-4057-a2c5-746c7a9b478f\") "
Nov 02 14:15:18.717948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.716402    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-maas-api-service-ca\") pod \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\" (UID: \"541c34f6-f73d-4cb6-9c91-ebb4b973e85d\") "
Nov 02 14:15:18.719351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.718860    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2bed0ec-be72-4057-a2c5-746c7a9b478f-maas-api-service-ca" pod "c2bed0ec-be72-4057-a2c5-746c7a9b478f" (UID: "c2bed0ec-be72-4057-a2c5-746c7a9b478f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:18.720033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.719986    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-maas-api-service-ca" pod "541c34f6-f73d-4cb6-9c91-ebb4b973e85d" (UID: "541c34f6-f73d-4cb6-9c91-ebb4b973e85d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:18.819138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.811020    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-kube-api-access-jg7lr" pod "541c34f6-f73d-4cb6-9c91-ebb4b973e85d" (UID: "541c34f6-f73d-4cb6-9c91-ebb4b973e85d"). InnerVolumeSpecName "kube-api-access-jg7lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:18.830904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.827116    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg7lr\" (UniqueName: \"kubernetes.io/projected/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-kube-api-access-jg7lr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:18.830904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.827157    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2bed0ec-be72-4057-a2c5-746c7a9b478f-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:18.830904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.827173    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541c34f6-f73d-4cb6-9c91-ebb4b973e85d-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:18.858061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.858027    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:18.881845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.875823    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"]
Nov 02 14:15:18.885118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.885075    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2bed0ec-be72-4057-a2c5-746c7a9b478f-kube-api-access-7fr4c" pod "c2bed0ec-be72-4057-a2c5-746c7a9b478f" (UID: "c2bed0ec-be72-4057-a2c5-746c7a9b478f"). InnerVolumeSpecName "kube-api-access-7fr4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:18.886095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.886067    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerName="cleanup"
Nov 02 14:15:18.886268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.886254    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerName="cleanup"
Nov 02 14:15:18.886366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.886354    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerName="cleanup"
Nov 02 14:15:18.886451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.886441    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerName="cleanup"
Nov 02 14:15:18.889866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.889842    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerName="cleanup"
Nov 02 14:15:18.890002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.889987    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerName="cleanup"
Nov 02 14:15:18.890132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.890120    5034 cpu_manager.go:414] "containerMap: removing container" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerName="cleanup"
Nov 02 14:15:18.890226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.890215    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerName="cleanup"
Nov 02 14:15:18.890310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.890300    5034 cpu_manager.go:414] "containerMap: removing container" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerName="cleanup"
Nov 02 14:15:18.890390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.890379    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerName="cleanup"
Nov 02 14:15:18.890493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.890483    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerName="cleanup"
Nov 02 14:15:18.905584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.905555    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerName="cleanup"
Nov 02 14:15:18.905969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.905951    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerName="cleanup"
Nov 02 14:15:18.906199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.906186    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerName="cleanup"
Nov 02 14:15:18.906344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.906331    5034 cpu_manager.go:414] "containerMap: removing container" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerName="cleanup"
Nov 02 14:15:18.906527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.906515    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerName="cleanup"
Nov 02 14:15:18.906654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.906642    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerName="cleanup"
Nov 02 14:15:18.906758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.906747    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerName="cleanup"
Nov 02 14:15:18.923001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.922946    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerName="cleanup"
Nov 02 14:15:18.923281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.923266    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerName="cleanup"
Nov 02 14:15:18.923414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.923403    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerName="cleanup"
Nov 02 14:15:18.923529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.923518    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerName="cleanup"
Nov 02 14:15:18.923630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.923619    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" containerName="cleanup"
Nov 02 14:15:18.923725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.923716    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" containerName="cleanup"
Nov 02 14:15:18.923855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.923844    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerName="cleanup"
Nov 02 14:15:18.923968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.923956    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerName="cleanup"
Nov 02 14:15:18.924074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.924063    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" containerName="cleanup"
Nov 02 14:15:18.939801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.930823    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b167495-f48d-4e8a-b684-685b5ef4d615-kube-api-access-g29g9\" (UniqueName: \"kubernetes.io/projected/5b167495-f48d-4e8a-b684-685b5ef4d615-kube-api-access-g29g9\") pod \"5b167495-f48d-4e8a-b684-685b5ef4d615\" (UID: \"5b167495-f48d-4e8a-b684-685b5ef4d615\") "
Nov 02 14:15:18.939801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.930934    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b167495-f48d-4e8a-b684-685b5ef4d615-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b167495-f48d-4e8a-b684-685b5ef4d615-maas-api-service-ca\") pod \"5b167495-f48d-4e8a-b684-685b5ef4d615\" (UID: \"5b167495-f48d-4e8a-b684-685b5ef4d615\") "
Nov 02 14:15:18.939801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.931662    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fr4c\" (UniqueName: \"kubernetes.io/projected/c2bed0ec-be72-4057-a2c5-746c7a9b478f-kube-api-access-7fr4c\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:18.943233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:18.943161    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b167495-f48d-4e8a-b684-685b5ef4d615-maas-api-service-ca" pod "5b167495-f48d-4e8a-b684-685b5ef4d615" (UID: "5b167495-f48d-4e8a-b684-685b5ef4d615"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:19.008152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.008121    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"]
Nov 02 14:15:19.016137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.010401    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.034468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.034427    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b167495-f48d-4e8a-b684-685b5ef4d615-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:19.073706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.064285    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-0d1ca7-openshift-default-dockercfg-9n4nr\""
Nov 02 14:15:19.087400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.087349    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj" event={"ID":"541c34f6-f73d-4cb6-9c91-ebb4b973e85d","Type":"ContainerDied","Data":"4585d4ea84b8a01ac8b7759a0780d4f1b4828c8994f66dddfe728a762c4e5983"}
Nov 02 14:15:19.087673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.087641    5034 scope.go:122] "RemoveContainer" containerID="f50d3d17b633f78de8d2a31dd88eef4950b42c77bea14a547c81b07d6289c8b7"
Nov 02 14:15:19.088599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.088566    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"
Nov 02 14:15:19.096154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.096089    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b167495-f48d-4e8a-b684-685b5ef4d615-kube-api-access-g29g9" pod "5b167495-f48d-4e8a-b684-685b5ef4d615" (UID: "5b167495-f48d-4e8a-b684-685b5ef4d615"). InnerVolumeSpecName "kube-api-access-g29g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:19.126456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.119918    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"
Nov 02 14:15:19.126456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.122518    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4" event={"ID":"5b167495-f48d-4e8a-b684-685b5ef4d615","Type":"ContainerDied","Data":"e182f7214988ae34250377aca7c883f221a68ebdecd5950e5ed81fc050f64381"}
Nov 02 14:15:19.126456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.123198    5034 scope.go:122] "RemoveContainer" containerID="9a4f36961158ff19a6d30b42c03d5b38245514e358c083d309d8641c6f8424cc"
Nov 02 14:15:19.132258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.132225    5034 generic.go:356] "Generic (PLEG): container finished" podID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerID="1b3a3dfcd577535a33cbb156a55d8a5531585924ab5653c4d909b2d4c4f26d93" exitCode=60
Nov 02 14:15:19.132547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.132525    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" event={"ID":"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e","Type":"ContainerDied","Data":"1b3a3dfcd577535a33cbb156a55d8a5531585924ab5653c4d909b2d4c4f26d93"}
Nov 02 14:15:19.133471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.133444    5034 scope.go:122] "RemoveContainer" containerID="1b3a3dfcd577535a33cbb156a55d8a5531585924ab5653c4d909b2d4c4f26d93"
Nov 02 14:15:19.134130 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:19.134106    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893815-s6tzl_odh-ai-gateway-infra(ed7a63da-d98b-4cc7-b666-4ce4f2eee86e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e"
Nov 02 14:15:19.148899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.148833    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"]
Nov 02 14:15:19.149964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.149946    5034 scope.go:122] "RemoveContainer" containerID="58f9f53d2767ee5cbf9a74d92e520ed02c41734fa161f105d65b8d2fd8f5b9a8"
Nov 02 14:15:19.151828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.151501    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup" containerID="cri-o://6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0" gracePeriod=30
Nov 02 14:15:19.151828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.151626    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"
Nov 02 14:15:19.160856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.152263    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6" event={"ID":"c2bed0ec-be72-4057-a2c5-746c7a9b478f","Type":"ContainerDied","Data":"c49fe973d6e3a158e145eb8a88d9bbaadf682f87a3c2ba2252c4b717e6c71f20"}
Nov 02 14:15:19.175237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175202    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-token\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.175410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175255    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-podinfo\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.175410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175338    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istiod-ca-cert\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.175410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175387    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-data\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.175581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175415    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc6s9\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-kube-api-access-zc6s9\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.175581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175445    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.175581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175475    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-certs\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.175581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175545    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-credential-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.175876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175703    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-envoy\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.175876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.175767    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g29g9\" (UniqueName: \"kubernetes.io/projected/5b167495-f48d-4e8a-b684-685b5ef4d615-kube-api-access-g29g9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:19.183458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.183432    5034 scope.go:122] "RemoveContainer" containerID="008eec60ec5edcf29ec2bc40e51639029675fdd3044743b1152faec55e0ed1db"
Nov 02 14:15:19.202483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.202440    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"]
Nov 02 14:15:19.212574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.212507    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"]
Nov 02 14:15:19.224652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.224464    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893815-d4szj"]
Nov 02 14:15:19.278481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.278433    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-credential-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.280126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.280090    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-envoy\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.280323    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-token\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.280360    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-podinfo\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.280444    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istiod-ca-cert\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.280512    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-data\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.280554    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc6s9\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-kube-api-access-zc6s9\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.280585    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.280631    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-certs\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.280658    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"]
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.281507    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-data\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.282167    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.282430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.282390    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istiod-ca-cert\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.283474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.283452    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-credential-socket\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.284695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.284082    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-envoy\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.284824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.284700    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-certs\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.284824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.284808    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-podinfo\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.314674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.313955    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc6s9\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-kube-api-access-zc6s9\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.347009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.346395    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-token\") pod \"e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") " pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.364083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.362913    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:15:19.388745 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.374280    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:15:19.418242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.418203    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:19.418510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.418487    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541c34f6-f73d-4cb6-9c91-ebb4b973e85d" path="/var/lib/kubelet/pods/541c34f6-f73d-4cb6-9c91-ebb4b973e85d/volumes"
Nov 02 14:15:19.423799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.423758    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:19.484419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.484135    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-maas-api-service-ca\") pod \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\" (UID: \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\") "
Nov 02 14:15:19.484419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.484276    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-kube-api-access-ps8nq\" (UniqueName: \"kubernetes.io/projected/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-kube-api-access-ps8nq\") pod \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\" (UID: \"95d3f836-bac2-42a5-8c33-8251fc8b9bc8\") "
Nov 02 14:15:19.494446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.487688    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-maas-api-service-ca" pod "95d3f836-bac2-42a5-8c33-8251fc8b9bc8" (UID: "95d3f836-bac2-42a5-8c33-8251fc8b9bc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:19.494601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.490371    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-kube-api-access-ps8nq" pod "95d3f836-bac2-42a5-8c33-8251fc8b9bc8" (UID: "95d3f836-bac2-42a5-8c33-8251fc8b9bc8"). InnerVolumeSpecName "kube-api-access-ps8nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:19.510466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.510355    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"]
Nov 02 14:15:19.518874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.516441    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-92f249-29893815-ndkf6"]
Nov 02 14:15:19.533118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.533090    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:19.586229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.586192    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42575ae9-31d2-4b23-8ab4-e0a3449dc406-kube-api-access-m6pcj\" (UniqueName: \"kubernetes.io/projected/42575ae9-31d2-4b23-8ab4-e0a3449dc406-kube-api-access-m6pcj\") pod \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\" (UID: \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\") "
Nov 02 14:15:19.586570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.586518    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42575ae9-31d2-4b23-8ab4-e0a3449dc406-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42575ae9-31d2-4b23-8ab4-e0a3449dc406-maas-api-service-ca\") pod \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\" (UID: \"42575ae9-31d2-4b23-8ab4-e0a3449dc406\") "
Nov 02 14:15:19.586983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.586956    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42575ae9-31d2-4b23-8ab4-e0a3449dc406-maas-api-service-ca" pod "42575ae9-31d2-4b23-8ab4-e0a3449dc406" (UID: "42575ae9-31d2-4b23-8ab4-e0a3449dc406"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:19.587286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.587265    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:19.587357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.587291    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42575ae9-31d2-4b23-8ab4-e0a3449dc406-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:19.587357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.587305    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps8nq\" (UniqueName: \"kubernetes.io/projected/95d3f836-bac2-42a5-8c33-8251fc8b9bc8-kube-api-access-ps8nq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:19.589376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.589346    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42575ae9-31d2-4b23-8ab4-e0a3449dc406-kube-api-access-m6pcj" pod "42575ae9-31d2-4b23-8ab4-e0a3449dc406" (UID: "42575ae9-31d2-4b23-8ab4-e0a3449dc406"). InnerVolumeSpecName "kube-api-access-m6pcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:19.632883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.632849    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"]
Nov 02 14:15:19.640718 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.640682    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-f30173-29893815-npcs4"]
Nov 02 14:15:19.688724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.688682    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6pcj\" (UniqueName: \"kubernetes.io/projected/42575ae9-31d2-4b23-8ab4-e0a3449dc406-kube-api-access-m6pcj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:19.788461 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:15:19.787570    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9889e75_036c_4358_a6c8_cbd4e7c0b0bc.slice/crio-70a9e09570197883330d5ca5cc1e4b5465c2593d04e80f937e145da45918e7b2 WatchSource:0}: Error finding container 70a9e09570197883330d5ca5cc1e4b5465c2593d04e80f937e145da45918e7b2: Status 404 returned error can't find the container with id 70a9e09570197883330d5ca5cc1e4b5465c2593d04e80f937e145da45918e7b2
Nov 02 14:15:19.789228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.789199    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"]
Nov 02 14:15:19.792012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.791955    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:19.792134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.792049    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:19.792134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:19.792097    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:20.168568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.168477    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" event={"ID":"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc","Type":"ContainerStarted","Data":"0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e"}
Nov 02 14:15:20.168758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.168588    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" event={"ID":"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc","Type":"ContainerStarted","Data":"70a9e09570197883330d5ca5cc1e4b5465c2593d04e80f937e145da45918e7b2"}
Nov 02 14:15:20.172378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.172343    5034 generic.go:356] "Generic (PLEG): container finished" podID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerID="6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0" exitCode=60
Nov 02 14:15:20.172522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.172425    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"
Nov 02 14:15:20.172522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.172439    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" event={"ID":"95d3f836-bac2-42a5-8c33-8251fc8b9bc8","Type":"ContainerDied","Data":"6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0"}
Nov 02 14:15:20.172640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.172573    5034 scope.go:122] "RemoveContainer" containerID="6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0"
Nov 02 14:15:20.172837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.172816    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6" event={"ID":"95d3f836-bac2-42a5-8c33-8251fc8b9bc8","Type":"ContainerDied","Data":"3b2d516f4a3acb29e876d65f2a7fcae8f21dfb4d3866a9710a54c592e873ba29"}
Nov 02 14:15:20.191807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.189653    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"
Nov 02 14:15:20.191807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.189673    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw" event={"ID":"42575ae9-31d2-4b23-8ab4-e0a3449dc406","Type":"ContainerDied","Data":"1af46068ccb1e9066023c30c6095e91acb1f9d993ffa92be122378872938069b"}
Nov 02 14:15:20.193743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.193721    5034 scope.go:122] "RemoveContainer" containerID="1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857"
Nov 02 14:15:20.211071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.211035    5034 generic.go:356] "Generic (PLEG): container finished" podID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerID="7905927e6c0541af2c67def2b90b229c007cd4c8469a76f6f2ee76a82f09b972" exitCode=60
Nov 02 14:15:20.211229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.211085    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" event={"ID":"ce6bd3ef-9a24-42bd-8372-2683bf34f30d","Type":"ContainerDied","Data":"7905927e6c0541af2c67def2b90b229c007cd4c8469a76f6f2ee76a82f09b972"}
Nov 02 14:15:20.211675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.211651    5034 scope.go:122] "RemoveContainer" containerID="7905927e6c0541af2c67def2b90b229c007cd4c8469a76f6f2ee76a82f09b972"
Nov 02 14:15:20.212333 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:20.212307    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9_odh-ai-gateway-infra(ce6bd3ef-9a24-42bd-8372-2683bf34f30d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d"
Nov 02 14:15:20.223611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.223586    5034 scope.go:122] "RemoveContainer" containerID="6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0"
Nov 02 14:15:20.224057 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:20.224021    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0\": container with ID starting with 6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0 not found: ID does not exist" containerID="6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0"
Nov 02 14:15:20.224166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.224061    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0"} err="failed to get container status \"6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0\": rpc error: code = NotFound desc = could not find container \"6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0\": container with ID starting with 6f55fb673ddcc1d3c999eb9b1c13cbdf1cee11c9d719d463c31fb139417f5ea0 not found: ID does not exist"
Nov 02 14:15:20.224166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.224088    5034 scope.go:122] "RemoveContainer" containerID="1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857"
Nov 02 14:15:20.224428 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:20.224407    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857\": container with ID starting with 1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857 not found: ID does not exist" containerID="1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857"
Nov 02 14:15:20.224474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.224438    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857"} err="failed to get container status \"1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857\": rpc error: code = NotFound desc = could not find container \"1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857\": container with ID starting with 1b1617d6a26a2a45016241dc44ccd6961f60dc357208b475d8f6e44cf0bc6857 not found: ID does not exist"
Nov 02 14:15:20.224474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.224461    5034 scope.go:122] "RemoveContainer" containerID="61316f92108b2432c8cdb9cf2afc555b8b60f2d6c22256b1a144698dae14cdc1"
Nov 02 14:15:20.230186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.230133    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podStartSLOduration=2.230114263 podStartE2EDuration="2.230114263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:20.220290894 +0000 UTC m=+1910.971595333" watchObservedRunningTime="2026-11-02 14:15:20.230114263 +0000 UTC m=+1910.981418701"
Nov 02 14:15:20.249601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.249570    5034 scope.go:122] "RemoveContainer" containerID="fa82bfd231c303fe7ac099bbe25a8ce6f8a68fdf007caf12bae16f59ce3dc0ba"
Nov 02 14:15:20.261026 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:20.260986    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42575ae9_31d2_4b23_8ab4_e0a3449dc406.slice/crio-1af46068ccb1e9066023c30c6095e91acb1f9d993ffa92be122378872938069b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42575ae9_31d2_4b23_8ab4_e0a3449dc406.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 14:15:20.277025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.276978    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"]
Nov 02 14:15:20.283365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.283296    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-ad42e0-29893815-928hw"]
Nov 02 14:15:20.311840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.311808    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"]
Nov 02 14:15:20.318672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.318636    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893815-dkrm6"]
Nov 02 14:15:20.419108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.419019    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:20.424908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.424881    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:20.425221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.425191    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:20.502384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.502345    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-kube-api-access-m8lcb\" (UniqueName: \"kubernetes.io/projected/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-kube-api-access-m8lcb\") pod \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\" (UID: \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\") "
Nov 02 14:15:20.502900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.502410    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-maas-api-service-ca\") pod \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\" (UID: \"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e\") "
Nov 02 14:15:20.502900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.502887    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-maas-api-service-ca" pod "ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" (UID: "ed7a63da-d98b-4cc7-b666-4ce4f2eee86e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:20.503375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.503351    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:20.504883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.504856    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-kube-api-access-m8lcb" pod "ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" (UID: "ed7a63da-d98b-4cc7-b666-4ce4f2eee86e"). InnerVolumeSpecName "kube-api-access-m8lcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:20.605379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:20.605335    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8lcb\" (UniqueName: \"kubernetes.io/projected/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e-kube-api-access-m8lcb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:21.228700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.228669    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"
Nov 02 14:15:21.228700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.228664    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl" event={"ID":"ed7a63da-d98b-4cc7-b666-4ce4f2eee86e","Type":"ContainerDied","Data":"5129d3a0d7a5c14143fb4466f69074ebdf7ce0b07a5179893f8ce9e922e44722"}
Nov 02 14:15:21.228977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.228744    5034 scope.go:122] "RemoveContainer" containerID="1b3a3dfcd577535a33cbb156a55d8a5531585924ab5653c4d909b2d4c4f26d93"
Nov 02 14:15:21.230079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.230059    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:21.231208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.231180    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:15:21.360160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.360014    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"]
Nov 02 14:15:21.364654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.364626    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893815-s6tzl"]
Nov 02 14:15:21.430472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.430426    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" path="/var/lib/kubelet/pods/42575ae9-31d2-4b23-8ab4-e0a3449dc406/volumes"
Nov 02 14:15:21.431420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.431394    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b167495-f48d-4e8a-b684-685b5ef4d615" path="/var/lib/kubelet/pods/5b167495-f48d-4e8a-b684-685b5ef4d615/volumes"
Nov 02 14:15:21.431871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.431856    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" path="/var/lib/kubelet/pods/95d3f836-bac2-42a5-8c33-8251fc8b9bc8/volumes"
Nov 02 14:15:21.432672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.432651    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2bed0ec-be72-4057-a2c5-746c7a9b478f" path="/var/lib/kubelet/pods/c2bed0ec-be72-4057-a2c5-746c7a9b478f/volumes"
Nov 02 14:15:21.433247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.433229    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" path="/var/lib/kubelet/pods/ed7a63da-d98b-4cc7-b666-4ce4f2eee86e/volumes"
Nov 02 14:15:21.434584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.434565    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"]
Nov 02 14:15:21.620484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.620448    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:21.723423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.723389    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-kube-api-access-5rd4l\" (UniqueName: \"kubernetes.io/projected/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-kube-api-access-5rd4l\") pod \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\" (UID: \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\") "
Nov 02 14:15:21.723729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.723710    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-maas-api-service-ca\") pod \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\" (UID: \"ce6bd3ef-9a24-42bd-8372-2683bf34f30d\") "
Nov 02 14:15:21.724224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.724134    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-maas-api-service-ca" pod "ce6bd3ef-9a24-42bd-8372-2683bf34f30d" (UID: "ce6bd3ef-9a24-42bd-8372-2683bf34f30d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:15:21.724577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.724556    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:21.726372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.726340    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-kube-api-access-5rd4l" pod "ce6bd3ef-9a24-42bd-8372-2683bf34f30d" (UID: "ce6bd3ef-9a24-42bd-8372-2683bf34f30d"). InnerVolumeSpecName "kube-api-access-5rd4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:15:21.826443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:21.826404    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rd4l\" (UniqueName: \"kubernetes.io/projected/ce6bd3ef-9a24-42bd-8372-2683bf34f30d-kube-api-access-5rd4l\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:15:22.241671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:22.241643    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"
Nov 02 14:15:22.241932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:22.241677    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9" event={"ID":"ce6bd3ef-9a24-42bd-8372-2683bf34f30d","Type":"ContainerDied","Data":"3af76d0cd5234c4abc7a3eb7fd03dba0aed9cf82206e9051d3c23aec801e02bf"}
Nov 02 14:15:22.241932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:22.241744    5034 scope.go:122] "RemoveContainer" containerID="7905927e6c0541af2c67def2b90b229c007cd4c8469a76f6f2ee76a82f09b972"
Nov 02 14:15:22.300532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:22.300492    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"]
Nov 02 14:15:22.306263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:22.306231    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893815-zrmn9"]
Nov 02 14:15:23.397138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:23.397106    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" path="/var/lib/kubelet/pods/ce6bd3ef-9a24-42bd-8372-2683bf34f30d/volumes"
Nov 02 14:15:24.716798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.713839    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"]
Nov 02 14:15:24.718461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.718429    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerName="cleanup"
Nov 02 14:15:24.718661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.718643    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerName="cleanup"
Nov 02 14:15:24.718814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.718800    5034 cpu_manager.go:414] "containerMap: removing container" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerName="cleanup"
Nov 02 14:15:24.718931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.718918    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerName="cleanup"
Nov 02 14:15:24.719036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719016    5034 cpu_manager.go:414] "containerMap: removing container" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup"
Nov 02 14:15:24.719137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719124    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup"
Nov 02 14:15:24.719241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719230    5034 cpu_manager.go:414] "containerMap: removing container" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup"
Nov 02 14:15:24.719326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719315    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup"
Nov 02 14:15:24.719410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719400    5034 cpu_manager.go:414] "containerMap: removing container" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerName="cleanup"
Nov 02 14:15:24.719486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719476    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerName="cleanup"
Nov 02 14:15:24.719580 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719570    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerName="cleanup"
Nov 02 14:15:24.719653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719643    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerName="cleanup"
Nov 02 14:15:24.719729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719720    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerName="cleanup"
Nov 02 14:15:24.719833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719820    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerName="cleanup"
Nov 02 14:15:24.719934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.719923    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerName="cleanup"
Nov 02 14:15:24.720014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.720002    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerName="cleanup"
Nov 02 14:15:24.720124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.720113    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerName="cleanup"
Nov 02 14:15:24.720207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.720196    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerName="cleanup"
Nov 02 14:15:24.720290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.720281    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerName="cleanup"
Nov 02 14:15:24.720360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.720351    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerName="cleanup"
Nov 02 14:15:24.720687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.720669    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup"
Nov 02 14:15:24.720816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.720803    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerName="cleanup"
Nov 02 14:15:24.720981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.720967    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerName="cleanup"
Nov 02 14:15:24.721085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.721074    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerName="cleanup"
Nov 02 14:15:24.721167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.721157    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerName="cleanup"
Nov 02 14:15:24.721250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.721238    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerName="cleanup"
Nov 02 14:15:24.721336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.721324    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7a63da-d98b-4cc7-b666-4ce4f2eee86e" containerName="cleanup"
Nov 02 14:15:24.721438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.721429    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerName="cleanup"
Nov 02 14:15:24.721526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.721515    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup"
Nov 02 14:15:24.726168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.726146    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"]
Nov 02 14:15:24.726421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.726402    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:24.753615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.751967    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-0d1ca7\""
Nov 02 14:15:24.774496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.760451    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-0d1ca7-dockercfg-hn2t9\""
Nov 02 14:15:24.794883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.791331    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8sp\" (UniqueName: \"kubernetes.io/projected/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-kube-api-access-rb8sp\") pod \"payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4\" (UID: \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:24.794883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.791409    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4\" (UID: \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:24.892956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.892916    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8sp\" (UniqueName: \"kubernetes.io/projected/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-kube-api-access-rb8sp\") pod \"payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4\" (UID: \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:24.893149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.892992    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4\" (UID: \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:24.893733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.893705    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4\" (UID: \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:24.918445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:24.918407    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8sp\" (UniqueName: \"kubernetes.io/projected/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-kube-api-access-rb8sp\") pod \"payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4\" (UID: \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:25.082711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:25.082675    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:25.417676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:25.417639    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"]
Nov 02 14:15:25.419347 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:15:25.419315    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f3d6f9_5cb7_44ea_8a61_1d7e9faf6938.slice/crio-7eb4cff1ec5bf49b316650d5bbefbc2b8368eb005fff7be173fd2d8088821a61 WatchSource:0}: Error finding container 7eb4cff1ec5bf49b316650d5bbefbc2b8368eb005fff7be173fd2d8088821a61: Status 404 returned error can't find the container with id 7eb4cff1ec5bf49b316650d5bbefbc2b8368eb005fff7be173fd2d8088821a61
Nov 02 14:15:26.294201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:26.294156    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4" event={"ID":"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938","Type":"ContainerStarted","Data":"cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242"}
Nov 02 14:15:26.294771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:26.294236    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:26.294771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:26.294260    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4" event={"ID":"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938","Type":"ContainerStarted","Data":"7eb4cff1ec5bf49b316650d5bbefbc2b8368eb005fff7be173fd2d8088821a61"}
Nov 02 14:15:26.339659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:26.339595    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4" podStartSLOduration=2.339575967 podStartE2EDuration="2.339575967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:26.337391528 +0000 UTC m=+1917.088695966" watchObservedRunningTime="2026-11-02 14:15:26.339575967 +0000 UTC m=+1917.090880406"
Nov 02 14:15:32.541799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.541670    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"]
Nov 02 14:15:32.542877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.542850    5034 cpu_manager.go:414] "containerMap: removing container" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerName="cleanup"
Nov 02 14:15:32.542877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.542878    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerName="cleanup"
Nov 02 14:15:32.543064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.542979    5034 cpu_manager.go:414] "containerMap: removing container" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup"
Nov 02 14:15:32.543064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.542989    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup"
Nov 02 14:15:32.543194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.543172    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6bd3ef-9a24-42bd-8372-2683bf34f30d" containerName="cleanup"
Nov 02 14:15:32.543254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.543206    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d3f836-bac2-42a5-8c33-8251fc8b9bc8" containerName="cleanup"
Nov 02 14:15:32.543254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.543218    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="42575ae9-31d2-4b23-8ab4-e0a3449dc406" containerName="cleanup"
Nov 02 14:15:32.555983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.555021    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.560013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.558769    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"]
Nov 02 14:15:32.581875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.577128    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-5cbbfb-openshift-default-dockercfg-rm7kc\""
Nov 02 14:15:32.741801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.741137    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istiod-ca-cert\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.741801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.741224    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-credential-socket\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.741801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.741255    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-envoy\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.741801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.741398    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-certs\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.741801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.741437    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-podinfo\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.741801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.741476    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsbj\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-kube-api-access-vwsbj\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.741801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.741549    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-token\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.741801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.741588    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-socket\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.741801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.741659    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-data\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843118    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istiod-ca-cert\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843206    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-credential-socket\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843234    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-envoy\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843383    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-certs\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843425    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-podinfo\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843465    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsbj\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-kube-api-access-vwsbj\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843543    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-token\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843584    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-socket\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843659    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-data\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843660    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-credential-socket\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.843961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.843905    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istiod-ca-cert\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.844258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.844236    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-certs\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.844379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.844317    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-data\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.844379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.844348    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-socket\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.846000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.845978    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-podinfo\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.846264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.846244    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-envoy\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.873814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.869155    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-token\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.874809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.874767    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsbj\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-kube-api-access-vwsbj\") pod \"e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") " pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:32.925432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:32.924598    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:33.329808 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:15:33.329764    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a55e6c_c9fb_40f0_a9c5_e8e9fd057dfa.slice/crio-ed1b338236d5679668f3fb3a3fe662e2eba761e13821ae27a72fff39883340a4 WatchSource:0}: Error finding container ed1b338236d5679668f3fb3a3fe662e2eba761e13821ae27a72fff39883340a4: Status 404 returned error can't find the container with id ed1b338236d5679668f3fb3a3fe662e2eba761e13821ae27a72fff39883340a4
Nov 02 14:15:33.331864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:33.331838    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"]
Nov 02 14:15:33.332606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:33.332575    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:33.332676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:33.332641    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:33.332676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:33.332672    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:33.467893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:33.467841    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" event={"ID":"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa","Type":"ContainerStarted","Data":"a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681"}
Nov 02 14:15:33.468198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:33.468173    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" event={"ID":"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa","Type":"ContainerStarted","Data":"ed1b338236d5679668f3fb3a3fe662e2eba761e13821ae27a72fff39883340a4"}
Nov 02 14:15:33.541857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:33.541768    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" podStartSLOduration=1.541744969 podStartE2EDuration="1.541744969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:33.533475364 +0000 UTC m=+1924.284779805" watchObservedRunningTime="2026-11-02 14:15:33.541744969 +0000 UTC m=+1924.293049413"
Nov 02 14:15:33.931273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:33.929006    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:34.931755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:34.931717    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:35.517334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.517303    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:35.518421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.518397    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:15:35.647717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.646841    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"]
Nov 02 14:15:35.654815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.654792    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.689413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.687932    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"]
Nov 02 14:15:35.704845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.692842    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fcefcc-openshift-default-dockercfg-r7kkx\""
Nov 02 14:15:35.712455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.710903    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.712455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.710976    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.712455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.711058    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.712455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.711234    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.712455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.711323    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdtdh\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-kube-api-access-gdtdh\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.712455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.711375    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.712455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.711412    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.712455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.711438    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.712455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.711470    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.813570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.813479    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.813570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.813552    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.813860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.813670    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.813860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.813705    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.813860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.813757    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.813860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.813857    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.814105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.813976    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdtdh\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-kube-api-access-gdtdh\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.814105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.814012    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.814105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.814033    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.814105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.814078    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.814308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.814136    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.814971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.814626    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.814971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.814922    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.814971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.814949    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.817379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.817355    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.818131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.818108    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.838502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.838479    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdtdh\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-kube-api-access-gdtdh\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.846516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.839184    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:35.995508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:35.995469    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:36.327419 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:15:36.327375    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b8e4bb_ded4_47cc_b577_8312a7a6b502.slice/crio-5676b844233fdd75c9ecae7392937ae67d8fae256aa9f8593ec585e7e21f4456 WatchSource:0}: Error finding container 5676b844233fdd75c9ecae7392937ae67d8fae256aa9f8593ec585e7e21f4456: Status 404 returned error can't find the container with id 5676b844233fdd75c9ecae7392937ae67d8fae256aa9f8593ec585e7e21f4456
Nov 02 14:15:36.329347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:36.329321    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"]
Nov 02 14:15:36.330867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:36.330835    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:36.330976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:36.330915    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:36.330976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:36.330961    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:36.530980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:36.530941    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" event={"ID":"d2b8e4bb-ded4-47cc-b577-8312a7a6b502","Type":"ContainerStarted","Data":"c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2"}
Nov 02 14:15:36.531623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:36.531594    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" event={"ID":"d2b8e4bb-ded4-47cc-b577-8312a7a6b502","Type":"ContainerStarted","Data":"5676b844233fdd75c9ecae7392937ae67d8fae256aa9f8593ec585e7e21f4456"}
Nov 02 14:15:36.585381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:36.585267    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" podStartSLOduration=1.585247525 podStartE2EDuration="1.585247525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:36.583051162 +0000 UTC m=+1927.334355601" watchObservedRunningTime="2026-11-02 14:15:36.585247525 +0000 UTC m=+1927.336551964"
Nov 02 14:15:36.996515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:36.996466    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:37.356236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:37.356168    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:15:37.964802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:37.956821    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"]
Nov 02 14:15:38.046800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.039474    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Nov 02 14:15:38.046800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.039531    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Nov 02 14:15:38.066799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.055871    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:15:38.137066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.133363    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfk42\" (UniqueName: \"kubernetes.io/projected/badd13e0-44f3-457e-b6f4-9d72352963a2-kube-api-access-qfk42\") pod \"payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj\" (UID: \"badd13e0-44f3-457e-b6f4-9d72352963a2\") " pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:15:38.137066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.133581    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badd13e0-44f3-457e-b6f4-9d72352963a2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj\" (UID: \"badd13e0-44f3-457e-b6f4-9d72352963a2\") " pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:15:38.137066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.133976    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5cbbfb\""
Nov 02 14:15:38.142372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.141665    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"]
Nov 02 14:15:38.197825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.178027    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5cbbfb-dockercfg-mp4jg\""
Nov 02 14:15:38.235105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.235075    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfk42\" (UniqueName: \"kubernetes.io/projected/badd13e0-44f3-457e-b6f4-9d72352963a2-kube-api-access-qfk42\") pod \"payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj\" (UID: \"badd13e0-44f3-457e-b6f4-9d72352963a2\") " pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:15:38.235343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.235319    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badd13e0-44f3-457e-b6f4-9d72352963a2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj\" (UID: \"badd13e0-44f3-457e-b6f4-9d72352963a2\") " pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:15:38.236037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.236018    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badd13e0-44f3-457e-b6f4-9d72352963a2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj\" (UID: \"badd13e0-44f3-457e-b6f4-9d72352963a2\") " pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:15:38.263493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.263460    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfk42\" (UniqueName: \"kubernetes.io/projected/badd13e0-44f3-457e-b6f4-9d72352963a2-kube-api-access-qfk42\") pod \"payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj\" (UID: \"badd13e0-44f3-457e-b6f4-9d72352963a2\") " pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:15:38.443927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.443890    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:15:38.803832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:38.803583    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"]
Nov 02 14:15:38.804663 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:15:38.804622    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadd13e0_44f3_457e_b6f4_9d72352963a2.slice/crio-655a8a4566e27cb22dd2315a0fca5b494faf2594e1f7919db5bee2f8161a7a43 WatchSource:0}: Error finding container 655a8a4566e27cb22dd2315a0fca5b494faf2594e1f7919db5bee2f8161a7a43: Status 404 returned error can't find the container with id 655a8a4566e27cb22dd2315a0fca5b494faf2594e1f7919db5bee2f8161a7a43
Nov 02 14:15:39.000807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:39.000226    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Nov 02 14:15:39.000807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:39.000280    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Nov 02 14:15:39.597440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:39.597398    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj" event={"ID":"badd13e0-44f3-457e-b6f4-9d72352963a2","Type":"ContainerStarted","Data":"b7ebaaf014d6462507ef86fc86e05fd3b5e47507977c5ee1c8ed8b7b0e156c31"}
Nov 02 14:15:39.602799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:39.598061    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj" event={"ID":"badd13e0-44f3-457e-b6f4-9d72352963a2","Type":"ContainerStarted","Data":"655a8a4566e27cb22dd2315a0fca5b494faf2594e1f7919db5bee2f8161a7a43"}
Nov 02 14:15:39.604297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:39.604272    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:15:39.677810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:39.669432    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj" podStartSLOduration=2.669410793 podStartE2EDuration="2.669410793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:15:39.659076385 +0000 UTC m=+1930.410380824" watchObservedRunningTime="2026-11-02 14:15:39.669410793 +0000 UTC m=+1930.420715233"
Nov 02 14:15:40.007837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:40.005019    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Nov 02 14:15:40.007837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:40.005075    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Nov 02 14:15:40.612174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:40.612139    5034 generic.go:356] "Generic (PLEG): container finished" podID="444082ae-6bfd-4126-8ae1-f30e98929081" containerID="c0e9bc5b929fd4d895707b2f47678c15e1cbb737b37e5cddce3d389d65cf59ac" exitCode=0
Nov 02 14:15:40.612683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:40.612214    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" event={"ID":"444082ae-6bfd-4126-8ae1-f30e98929081","Type":"ContainerDied","Data":"c0e9bc5b929fd4d895707b2f47678c15e1cbb737b37e5cddce3d389d65cf59ac"}
Nov 02 14:15:40.612945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:40.612916    5034 scope.go:122] "RemoveContainer" containerID="c0e9bc5b929fd4d895707b2f47678c15e1cbb737b37e5cddce3d389d65cf59ac"
Nov 02 14:15:40.614748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:40.614719    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:40.614871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:40.614799    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:40.614871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:40.614834    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:15:41.000800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:41.000096    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Nov 02 14:15:41.000800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:41.000158    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Nov 02 14:15:41.693800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:41.693006    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" event={"ID":"444082ae-6bfd-4126-8ae1-f30e98929081","Type":"ContainerStarted","Data":"7adad05e0c6fb77909636bc4d73bede1c7c87e37f50e80dca2792b4cdad46079"}
Nov 02 14:15:41.801801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:41.792493    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:15:41.819805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:41.809754    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 14:15:41.820900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:41.820099    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 14:15:42.001212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:42.001171    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body=
Nov 02 14:15:42.002037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:42.001997    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused"
Nov 02 14:15:42.760496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:42.760458    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 14:15:42.761021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:42.760512    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 14:15:42.960085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:42.960047    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:15:43.316387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:43.316358    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:43.317101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:43.317076    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:43.318391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:43.318372    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:15:43.760867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:43.760835    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 14:15:43.761284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:43.760894    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 14:15:44.761152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:44.761117    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 14:15:44.761657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:44.761182    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 14:15:45.761131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:45.761100    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 14:15:45.761320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:45.761150    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 14:15:46.760937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:46.760902    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused" start-of-body=
Nov 02 14:15:46.761126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:46.760968    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.224:15021/healthz/ready\": dial tcp 10.132.0.224:15021: connect: connection refused"
Nov 02 14:15:47.765515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:47.765485    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:15:47.766677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:47.766657    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:15:48.416708 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:15:48.416670    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:15:51.698033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:15:51.698005    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:16:02.325202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.325155    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"]
Nov 02 14:16:02.335483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.334034    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.362697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.362668    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"]
Nov 02 14:16:02.376428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.374697    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-960a68\"/\"default-dockercfg-bhkht\""
Nov 02 14:16:02.376428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.375387    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-960a68\"/\"auth-test-model-960a68-kserve-self-signed-certs\""
Nov 02 14:16:02.376428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.375487    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-960a68\"/\"openshift-service-ca.crt\""
Nov 02 14:16:02.376428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.375555    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-960a68\"/\"kube-root-ca.crt\""
Nov 02 14:16:02.463695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.463655    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee64526-add5-4d0b-a53f-f420ba83d8ba-tls-certs\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.463945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.463847    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-home\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.463945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.463927    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-tmp-dir\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.464098 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.464045    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-dshm\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.464098 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.464069    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-model-cache\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.464229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.464104    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btlqk\" (UniqueName: \"kubernetes.io/projected/bee64526-add5-4d0b-a53f-f420ba83d8ba-kube-api-access-btlqk\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.565419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.565383    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-dshm\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.565602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.565426    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-model-cache\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.565602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.565464    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btlqk\" (UniqueName: \"kubernetes.io/projected/bee64526-add5-4d0b-a53f-f420ba83d8ba-kube-api-access-btlqk\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.565713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.565693    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee64526-add5-4d0b-a53f-f420ba83d8ba-tls-certs\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.565850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.565829    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-home\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.565937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.565879    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-model-cache\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.565937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.565889    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-tmp-dir\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.566120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.566101    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-tmp-dir\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.566252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.566223    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-home\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.568080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.568053    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-dshm\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.568627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.568607    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee64526-add5-4d0b-a53f-f420ba83d8ba-tls-certs\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.590488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.590412    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btlqk\" (UniqueName: \"kubernetes.io/projected/bee64526-add5-4d0b-a53f-f420ba83d8ba-kube-api-access-btlqk\") pod \"auth-test-model-960a68-kserve-ddbc55bc7-4xjrf\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") " pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:02.725438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:02.725088    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:03.063824 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:16:03.063789    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbee64526_add5_4d0b_a53f_f420ba83d8ba.slice/crio-1a333beb60f74f487ce40a18a36cf30d9452c998bbcb1dfd9ae0f7e508f64893 WatchSource:0}: Error finding container 1a333beb60f74f487ce40a18a36cf30d9452c998bbcb1dfd9ae0f7e508f64893: Status 404 returned error can't find the container with id 1a333beb60f74f487ce40a18a36cf30d9452c998bbcb1dfd9ae0f7e508f64893
Nov 02 14:16:03.066746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:03.066720    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"]
Nov 02 14:16:03.995626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:03.995584    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf" event={"ID":"bee64526-add5-4d0b-a53f-f420ba83d8ba","Type":"ContainerStarted","Data":"0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5"}
Nov 02 14:16:03.996161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:03.995635    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf" event={"ID":"bee64526-add5-4d0b-a53f-f420ba83d8ba","Type":"ContainerStarted","Data":"1a333beb60f74f487ce40a18a36cf30d9452c998bbcb1dfd9ae0f7e508f64893"}
Nov 02 14:16:04.043988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:04.043934    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf" podStartSLOduration=2.043920208 podStartE2EDuration="2.043920208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:04.039145757 +0000 UTC m=+1954.790450195" watchObservedRunningTime="2026-11-02 14:16:04.043920208 +0000 UTC m=+1954.795224649"
Nov 02 14:16:04.727491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:04.727457    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:05.007967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.007932    5034 generic.go:356] "Generic (PLEG): container finished" podID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerID="8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f" exitCode=0
Nov 02 14:16:05.008537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.008005    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" event={"ID":"3c2a56dd-c87f-4e98-91c8-e628aac57f7b","Type":"ContainerDied","Data":"8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f"}
Nov 02 14:16:05.008632 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.008613    5034 scope.go:122] "RemoveContainer" containerID="8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f"
Nov 02 14:16:05.010211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.010185    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:05.010323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.010252    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:05.010323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.010285    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:05.726330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.726295    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"]
Nov 02 14:16:05.731507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.731485    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.781801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.778871    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.781801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.778977    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.781801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.779069    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.781801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.779160    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.781801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.779213    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.781801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.779256    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.781801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.779347    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42zkx\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-kube-api-access-42zkx\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.781801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.779390    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.781801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.779452    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.818958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.818916    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"]
Nov 02 14:16:05.894808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.893906    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.894808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.893985    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.894808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.894030    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.894808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.894120    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zkx\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-kube-api-access-42zkx\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.894808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.894161    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.894808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.894223    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.894808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.894356    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.894808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.894447    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.894808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.894531    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.907565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.907529    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.908755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.908174    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.908932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.908903    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.909005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.908677    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.909005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.908744    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.909124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.908506    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.910540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.910513    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.939272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.939241    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:05.939402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:05.939387    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zkx\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-kube-api-access-42zkx\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:06.023377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.023343    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" event={"ID":"3c2a56dd-c87f-4e98-91c8-e628aac57f7b","Type":"ContainerStarted","Data":"f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9"}
Nov 02 14:16:06.047569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.047540    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:06.245767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.245741    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:16:06.247163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.247122    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 02 14:16:06.247325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.247184    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 02 14:16:06.304578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.304282    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:16:06.418380 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:16:06.418352    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b3c337c_f1fb_44cb_856d_1f86fc8d908b.slice/crio-30b442c0cc5e28d4131938ae0a08d8d36044795ed482a24f39eaec324f6009cc WatchSource:0}: Error finding container 30b442c0cc5e28d4131938ae0a08d8d36044795ed482a24f39eaec324f6009cc: Status 404 returned error can't find the container with id 30b442c0cc5e28d4131938ae0a08d8d36044795ed482a24f39eaec324f6009cc
Nov 02 14:16:06.421061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.421033    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"]
Nov 02 14:16:06.421536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.421505    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:06.421583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.421569    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:06.421624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:06.421601    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:07.034200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:07.034168    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" event={"ID":"1b3c337c-f1fb-44cb-856d-1f86fc8d908b","Type":"ContainerStarted","Data":"a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004"}
Nov 02 14:16:07.034697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:07.034225    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" event={"ID":"1b3c337c-f1fb-44cb-856d-1f86fc8d908b","Type":"ContainerStarted","Data":"30b442c0cc5e28d4131938ae0a08d8d36044795ed482a24f39eaec324f6009cc"}
Nov 02 14:16:07.047792 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:07.047759    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:07.049239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:07.049216    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Nov 02 14:16:07.049351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:07.049257    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Nov 02 14:16:07.110576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:07.110512    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" podStartSLOduration=2.110491996 podStartE2EDuration="2.110491996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:16:07.091827598 +0000 UTC m=+1957.843132038" watchObservedRunningTime="2026-11-02 14:16:07.110491996 +0000 UTC m=+1957.861796437"
Nov 02 14:16:07.247408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:07.246910    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 02 14:16:07.247408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:07.246994    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 02 14:16:08.049884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:08.049831    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Nov 02 14:16:08.050518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:08.050461    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Nov 02 14:16:08.246077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:08.246037    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 02 14:16:08.246314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:08.246293    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 02 14:16:09.048801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:09.048706    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Nov 02 14:16:09.049036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:09.048879    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Nov 02 14:16:09.246124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:09.246082    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 02 14:16:09.246515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:09.246138    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 02 14:16:10.058855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:10.058539    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Nov 02 14:16:10.058855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:10.058596    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Nov 02 14:16:10.249418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:10.248999    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 02 14:16:10.249418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:10.249058    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 02 14:16:11.050069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:11.049563    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Nov 02 14:16:11.050069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:11.049628    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Nov 02 14:16:11.245661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:11.245626    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused" start-of-body=
Nov 02 14:16:11.245970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:11.245689    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.225:15021/healthz/ready\": dial tcp 10.132.0.225:15021: connect: connection refused"
Nov 02 14:16:12.048277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:12.048243    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Nov 02 14:16:12.048745 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:12.048313    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Nov 02 14:16:12.278566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:12.278526    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:16:12.289096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:12.289063    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:16:12.734406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:12.734374    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:12.752213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:12.751980    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:13.049049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:13.048963    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body=
Nov 02 14:16:13.049049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:13.049025    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused"
Nov 02 14:16:13.221487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:13.221434    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:16:14.053914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:14.053884    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:14.054444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:14.054424    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:14.055150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:14.055127    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:16:18.414827 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:16:18.414794    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:16:19.054337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:19.054291    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"]
Nov 02 14:16:19.066396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:19.066365    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a75e93-gld2z"]
Nov 02 14:16:19.397053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:19.396969    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978b6664-2b85-4ec1-99d8-039e3d12fc69" path="/var/lib/kubelet/pods/978b6664-2b85-4ec1-99d8-039e3d12fc69/volumes"
Nov 02 14:16:21.814542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:21.814508    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": read tcp 10.132.0.2:50942->10.132.0.236:15021: read: connection reset by peer" start-of-body=
Nov 02 14:16:21.815206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:21.815153    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": read tcp 10.132.0.2:50942->10.132.0.236:15021: read: connection reset by peer"
Nov 02 14:16:22.326220 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:22.326189    5034 generic.go:356] "Generic (PLEG): container finished" podID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerID="0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e" exitCode=0
Nov 02 14:16:22.326416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:22.326258    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" event={"ID":"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc","Type":"ContainerDied","Data":"0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e"}
Nov 02 14:16:22.326764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:22.326749    5034 scope.go:122] "RemoveContainer" containerID="0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e"
Nov 02 14:16:22.328465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:22.328439    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:22.328529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:22.328505    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:22.328565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:22.328535    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:23.338723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:23.338687    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" event={"ID":"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc","Type":"ContainerStarted","Data":"78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734"}
Nov 02 14:16:23.420842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:23.418699    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Nov 02 14:16:23.420842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:23.418754    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Nov 02 14:16:23.423586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:23.422797    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:16:24.427807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:24.421164    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Nov 02 14:16:24.427807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:24.421288    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Nov 02 14:16:25.419101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:25.419062    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Nov 02 14:16:25.419331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:25.419129    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Nov 02 14:16:26.419216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:26.419176    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Nov 02 14:16:26.419736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:26.419236    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Nov 02 14:16:27.419569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:27.419531    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Nov 02 14:16:27.420039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:27.419588    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Nov 02 14:16:28.420073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:28.419615    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused" start-of-body=
Nov 02 14:16:28.420073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:28.419682    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.236:15021/healthz/ready\": dial tcp 10.132.0.236:15021: connect: connection refused"
Nov 02 14:16:29.487259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:29.487223    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:16:29.489803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:29.489759    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:16:29.490671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:29.490621    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:16:36.558940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:36.558895    5034 patch_prober.go:29] interesting pod/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:16:36.559512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:36.558955    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:16:39.609152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:39.609111    5034 generic.go:356] "Generic (PLEG): container finished" podID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerID="a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681" exitCode=0
Nov 02 14:16:39.609698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:39.609180    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" event={"ID":"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa","Type":"ContainerDied","Data":"a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681"}
Nov 02 14:16:39.609823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:39.609722    5034 scope.go:122] "RemoveContainer" containerID="a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681"
Nov 02 14:16:39.611747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:39.611713    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:39.611860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:39.611821    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:39.611924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:39.611869    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:16:40.623597 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:40.623563    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" event={"ID":"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa","Type":"ContainerStarted","Data":"869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0"}
Nov 02 14:16:40.925460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:40.925379    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:16:40.925638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:40.925587    5034 patch_prober.go:29] interesting pod/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 14:16:40.925723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:40.925646    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 14:16:41.926044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:41.926009    5034 patch_prober.go:29] interesting pod/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 14:16:41.926444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:41.926061    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 14:16:42.034686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:42.034654    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"]
Nov 02 14:16:42.047323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:42.047299    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4dbe85e-p9gsr"]
Nov 02 14:16:42.925298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:42.925261    5034 patch_prober.go:29] interesting pod/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 14:16:42.925493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:42.925336    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 14:16:43.411371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:43.411334    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e23929a-567b-4366-951d-d0fc22ca5b08" path="/var/lib/kubelet/pods/5e23929a-567b-4366-951d-d0fc22ca5b08/volumes"
Nov 02 14:16:43.925488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:43.925455    5034 patch_prober.go:29] interesting pod/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 14:16:43.925680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:43.925518    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 14:16:44.925654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:44.925620    5034 patch_prober.go:29] interesting pod/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 14:16:44.926111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:44.925670    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 14:16:45.040565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:45.040527    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"]
Nov 02 14:16:45.053463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:45.053433    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a4e90dd-5r4m2"]
Nov 02 14:16:45.408214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:45.408179    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e4cd2b-d057-43f2-91ce-fe2da8a72ec2" path="/var/lib/kubelet/pods/82e4cd2b-d057-43f2-91ce-fe2da8a72ec2/volumes"
Nov 02 14:16:45.925687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:45.925653    5034 patch_prober.go:29] interesting pod/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body=
Nov 02 14:16:45.926227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:45.925717    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused"
Nov 02 14:16:46.930324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:46.930287    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:16:46.930732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:46.930684    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:16:46.931314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:46.931294    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:16:48.418539 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:16:48.418494    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:16:55.032433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:55.032388    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"]
Nov 02 14:16:55.045427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:55.045401    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929a74eb-f9gdb"]
Nov 02 14:16:55.397142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:16:55.397062    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef83544-2544-4d8b-9089-127607007b15" path="/var/lib/kubelet/pods/bef83544-2544-4d8b-9089-127607007b15/volumes"
Nov 02 14:17:07.487487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.487454    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"]
Nov 02 14:17:07.492518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.492494    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.520234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.519158    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"]
Nov 02 14:17:07.522491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.522467    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-fcefcc\"/\"kube-root-ca.crt\""
Nov 02 14:17:07.522622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.522541    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-fcefcc\"/\"openshift-service-ca.crt\""
Nov 02 14:17:07.524611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.524588    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-fcefcc\"/\"default-dockercfg-q7f7j\""
Nov 02 14:17:07.524748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.524642    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-fcefcc\"/\"auth-test-model-fcefcc-kserve-self-signed-certs\""
Nov 02 14:17:07.611115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.611083    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdq2\" (UniqueName: \"kubernetes.io/projected/96422e8c-4df5-48aa-8b50-b871af7fddd7-kube-api-access-csdq2\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.611303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.611137    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-tmp-dir\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.611303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.611299    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96422e8c-4df5-48aa-8b50-b871af7fddd7-tls-certs\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.611425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.611396    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-model-cache\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.611484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.611440    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-dshm\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.611553 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.611539    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-home\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.731804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.717804    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-home\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.732646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.732619    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdq2\" (UniqueName: \"kubernetes.io/projected/96422e8c-4df5-48aa-8b50-b871af7fddd7-kube-api-access-csdq2\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.733237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.733216    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-tmp-dir\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.741839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.733837    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96422e8c-4df5-48aa-8b50-b871af7fddd7-tls-certs\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.743811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.743765    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-model-cache\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.760012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.759984    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-dshm\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.761699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.744223    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-model-cache\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.761889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.733642    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-tmp-dir\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.761996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.732510    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-home\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.762085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.761043    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96422e8c-4df5-48aa-8b50-b871af7fddd7-tls-certs\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.776152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.776121    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-dshm\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.782702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.782669    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdq2\" (UniqueName: \"kubernetes.io/projected/96422e8c-4df5-48aa-8b50-b871af7fddd7-kube-api-access-csdq2\") pod \"auth-test-model-fcefcc-kserve-64b9f56f-qkxf8\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") " pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:07.816771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:07.816739    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:08.036980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:08.036906    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"]
Nov 02 14:17:08.041658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:08.041630    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b7289c8-6k2cd"]
Nov 02 14:17:08.148892 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:17:08.148862    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96422e8c_4df5_48aa_8b50_b871af7fddd7.slice/crio-46d1414958e52c044a1171c2cd02f0dc684de1d55c2f66ac4a8d37addb1152fd WatchSource:0}: Error finding container 46d1414958e52c044a1171c2cd02f0dc684de1d55c2f66ac4a8d37addb1152fd: Status 404 returned error can't find the container with id 46d1414958e52c044a1171c2cd02f0dc684de1d55c2f66ac4a8d37addb1152fd
Nov 02 14:17:08.151252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:08.151228    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"]
Nov 02 14:17:08.883444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:08.883408    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8" event={"ID":"96422e8c-4df5-48aa-8b50-b871af7fddd7","Type":"ContainerStarted","Data":"9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5"}
Nov 02 14:17:08.883947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:08.883457    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8" event={"ID":"96422e8c-4df5-48aa-8b50-b871af7fddd7","Type":"ContainerStarted","Data":"46d1414958e52c044a1171c2cd02f0dc684de1d55c2f66ac4a8d37addb1152fd"}
Nov 02 14:17:08.932402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:08.932344    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8" podStartSLOduration=1.932331166 podStartE2EDuration="1.932331166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:17:08.929378191 +0000 UTC m=+2019.680682626" watchObservedRunningTime="2026-11-02 14:17:08.932331166 +0000 UTC m=+2019.683635603"
Nov 02 14:17:09.398941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:09.398906    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b54f09-2788-4fa9-8903-24c0aae57d4b" path="/var/lib/kubelet/pods/a9b54f09-2788-4fa9-8903-24c0aae57d4b/volumes"
Nov 02 14:17:09.817690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:09.817647    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:17.817926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:17.817824    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:17.831204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:17.831180    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:17.992998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:17.992972    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:17:18.415700 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:17:18.415663    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:17:21.043519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:21.043484    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"]
Nov 02 14:17:21.056964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:21.056934    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdecd625-5k2jh"]
Nov 02 14:17:21.396722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:21.396649    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6c3549-effa-43d5-b052-28f1d8800600" path="/var/lib/kubelet/pods/2e6c3549-effa-43d5-b052-28f1d8800600/volumes"
Nov 02 14:17:22.054521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:22.054428    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"]
Nov 02 14:17:22.066096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:22.066067    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f238028e-5xqvj"]
Nov 02 14:17:23.398569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:23.398530    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc0c34a-eafe-4374-a00c-29973601d7ed" path="/var/lib/kubelet/pods/5cc0c34a-eafe-4374-a00c-29973601d7ed/volumes"
Nov 02 14:17:31.050965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:31.050926    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"]
Nov 02 14:17:31.064359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:31.064328    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9056b5d3-rbs2s"]
Nov 02 14:17:31.419192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:31.419102    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8731eb60-f299-4c3c-8224-1b4a5f57a94c" path="/var/lib/kubelet/pods/8731eb60-f299-4c3c-8224-1b4a5f57a94c/volumes"
Nov 02 14:17:32.066803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:32.062846    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"]
Nov 02 14:17:32.079058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:32.079020    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a034a743-d5n6v"]
Nov 02 14:17:33.419800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:33.405384    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f0e18b7-26ee-46a7-aaf2-d16f46a25f56" path="/var/lib/kubelet/pods/2f0e18b7-26ee-46a7-aaf2-d16f46a25f56/volumes"
Nov 02 14:17:35.957528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:35.957497    5034 scope.go:122] "RemoveContainer" containerID="90ce572d55bffb4a6b0bfe885921f862e141332cb3413a89562b0706d520d6f5"
Nov 02 14:17:44.035464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:44.035427    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"]
Nov 02 14:17:44.050468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:44.050421    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af617526-7528d"]
Nov 02 14:17:45.396040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:45.395997    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2209c59-eb7c-4b99-adc1-a0133f6ca5b6" path="/var/lib/kubelet/pods/e2209c59-eb7c-4b99-adc1-a0133f6ca5b6/volumes"
Nov 02 14:17:48.415056 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:17:48.415023    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:17:49.037322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:49.037283    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"]
Nov 02 14:17:49.043591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:49.043564    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf725bc3-684qv"]
Nov 02 14:17:49.407017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:49.406933    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862142b3-9c3f-4315-8bc8-ed798b3cc675" path="/var/lib/kubelet/pods/862142b3-9c3f-4315-8bc8-ed798b3cc675/volumes"
Nov 02 14:17:51.032890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:51.032846    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"]
Nov 02 14:17:51.045580 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:51.045546    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29349365-6f5hz"]
Nov 02 14:17:51.399424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:51.399335    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f91d65d-6e57-403b-91e3-cb641250aa69" path="/var/lib/kubelet/pods/4f91d65d-6e57-403b-91e3-cb641250aa69/volumes"
Nov 02 14:17:53.313874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:53.313834    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"]
Nov 02 14:17:58.315282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.315218    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj" podUID="badd13e0-44f3-457e-b6f4-9d72352963a2" containerName="payload-processing" containerID="cri-o://b7ebaaf014d6462507ef86fc86e05fd3b5e47507977c5ee1c8ed8b7b0e156c31" gracePeriod=25
Nov 02 14:17:58.458184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.458154    5034 generic.go:356] "Generic (PLEG): container finished" podID="badd13e0-44f3-457e-b6f4-9d72352963a2" containerID="b7ebaaf014d6462507ef86fc86e05fd3b5e47507977c5ee1c8ed8b7b0e156c31" exitCode=0
Nov 02 14:17:58.458343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.458192    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj" event={"ID":"badd13e0-44f3-457e-b6f4-9d72352963a2","Type":"ContainerDied","Data":"b7ebaaf014d6462507ef86fc86e05fd3b5e47507977c5ee1c8ed8b7b0e156c31"}
Nov 02 14:17:58.638591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.638560    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:17:58.756925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.756496    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badd13e0-44f3-457e-b6f4-9d72352963a2-kube-api-access-qfk42\" (UniqueName: \"kubernetes.io/projected/badd13e0-44f3-457e-b6f4-9d72352963a2-kube-api-access-qfk42\") pod \"badd13e0-44f3-457e-b6f4-9d72352963a2\" (UID: \"badd13e0-44f3-457e-b6f4-9d72352963a2\") "
Nov 02 14:17:58.757495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.757463    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badd13e0-44f3-457e-b6f4-9d72352963a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badd13e0-44f3-457e-b6f4-9d72352963a2-plugins-config-volume\") pod \"badd13e0-44f3-457e-b6f4-9d72352963a2\" (UID: \"badd13e0-44f3-457e-b6f4-9d72352963a2\") "
Nov 02 14:17:58.758049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.758022    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badd13e0-44f3-457e-b6f4-9d72352963a2-plugins-config-volume" pod "badd13e0-44f3-457e-b6f4-9d72352963a2" (UID: "badd13e0-44f3-457e-b6f4-9d72352963a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:17:58.758385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.758364    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badd13e0-44f3-457e-b6f4-9d72352963a2-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:17:58.759053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.759027    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badd13e0-44f3-457e-b6f4-9d72352963a2-kube-api-access-qfk42" pod "badd13e0-44f3-457e-b6f4-9d72352963a2" (UID: "badd13e0-44f3-457e-b6f4-9d72352963a2"). InnerVolumeSpecName "kube-api-access-qfk42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:17:58.860062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:58.859981    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfk42\" (UniqueName: \"kubernetes.io/projected/badd13e0-44f3-457e-b6f4-9d72352963a2-kube-api-access-qfk42\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:17:59.470793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:59.470751    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"
Nov 02 14:17:59.471259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:59.470773    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj" event={"ID":"badd13e0-44f3-457e-b6f4-9d72352963a2","Type":"ContainerDied","Data":"655a8a4566e27cb22dd2315a0fca5b494faf2594e1f7919db5bee2f8161a7a43"}
Nov 02 14:17:59.471259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:59.470867    5034 scope.go:122] "RemoveContainer" containerID="b7ebaaf014d6462507ef86fc86e05fd3b5e47507977c5ee1c8ed8b7b0e156c31"
Nov 02 14:17:59.532424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:59.532388    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"]
Nov 02 14:17:59.544094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:17:59.544063    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5cbbfb-69bc46d765-4lvbj"]
Nov 02 14:18:01.397905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:01.397865    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badd13e0-44f3-457e-b6f4-9d72352963a2" path="/var/lib/kubelet/pods/badd13e0-44f3-457e-b6f4-9d72352963a2/volumes"
Nov 02 14:18:02.043233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:02.043190    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"]
Nov 02 14:18:02.054588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:02.054549    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af10ada4-zx9qn"]
Nov 02 14:18:03.395811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:03.395753    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76a7105-dc24-4bfc-ac67-09d4a1212ff7" path="/var/lib/kubelet/pods/b76a7105-dc24-4bfc-ac67-09d4a1212ff7/volumes"
Nov 02 14:18:10.035545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:10.035504    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"]
Nov 02 14:18:10.048662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:10.048631    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c5524fc-cs782"]
Nov 02 14:18:10.052868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:10.052839    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"]
Nov 02 14:18:10.056388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:10.056367    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70ff89b-d6mhx"]
Nov 02 14:18:11.037210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:11.037179    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"]
Nov 02 14:18:11.051082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:11.051055    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48377d11-mlllh"]
Nov 02 14:18:11.397887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:11.397804    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28827380-d430-4957-a4ad-ab59e6c4646f" path="/var/lib/kubelet/pods/28827380-d430-4957-a4ad-ab59e6c4646f/volumes"
Nov 02 14:18:11.398440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:11.398416    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d80e0281-ec3a-4ad1-a29f-262ccf7a408e" path="/var/lib/kubelet/pods/d80e0281-ec3a-4ad1-a29f-262ccf7a408e/volumes"
Nov 02 14:18:11.398971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:11.398951    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55d1057-cc5f-4e2f-8c1e-943223596e1e" path="/var/lib/kubelet/pods/f55d1057-cc5f-4e2f-8c1e-943223596e1e/volumes"
Nov 02 14:18:18.416941 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:18:18.416901    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:18:19.867544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:19.867501    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"]
Nov 02 14:18:19.868049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:19.867816    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy" containerID="cri-o://869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0" gracePeriod=30
Nov 02 14:18:21.016444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.016407    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"]
Nov 02 14:18:21.017466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.017446    5034 cpu_manager.go:414] "containerMap: removing container" podUID="badd13e0-44f3-457e-b6f4-9d72352963a2" containerName="payload-processing"
Nov 02 14:18:21.017577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.017468    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badd13e0-44f3-457e-b6f4-9d72352963a2" containerName="payload-processing"
Nov 02 14:18:21.017723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.017710    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="badd13e0-44f3-457e-b6f4-9d72352963a2" containerName="payload-processing"
Nov 02 14:18:21.023759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.023730    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.053977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.053945    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"]
Nov 02 14:18:21.083620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.083567    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-credential-socket\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.083988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.083962    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-podinfo\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.084387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.084364    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istiod-ca-cert\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.084731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.084689    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-envoy\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.085065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.085043    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5x76\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-kube-api-access-v5x76\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.085348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.085302    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-token\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.085586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.085567    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-data\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.085863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.085842    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-socket\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.086159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.086139    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-certs\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.087847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.087828    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-abc0f7-openshift-default-dockercfg-575pk\""
Nov 02 14:18:21.209806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.208717    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istiod-ca-cert\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.252808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.245593    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istiod-ca-cert\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.273889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.272401    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-envoy\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.273889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.272514    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5x76\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-kube-api-access-v5x76\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.273889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.272555    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-token\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.273889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.272608    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-data\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.273889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.272653    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-socket\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.273889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.272809    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-certs\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.273889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.272926    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-credential-socket\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.273889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.272986    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-podinfo\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.274481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.273963    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-data\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.274553 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.274501    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-certs\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.291835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.274733    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-socket\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.291835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.274962    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-credential-socket\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.294817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.294726    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-envoy\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.315461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.315424    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-podinfo\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.318715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.318648    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5x76\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-kube-api-access-v5x76\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.321319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.321277    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-token\") pod \"e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") " pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.345470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.344103    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:21.703168 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:18:21.703133    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f3ee52_861f_4ea2_8a4d_2148fae634c2.slice/crio-bb674fbb68a8cf845928c337795a8d2e2f7ce7e06094a19796cdda09f826b3e5 WatchSource:0}: Error finding container bb674fbb68a8cf845928c337795a8d2e2f7ce7e06094a19796cdda09f826b3e5: Status 404 returned error can't find the container with id bb674fbb68a8cf845928c337795a8d2e2f7ce7e06094a19796cdda09f826b3e5
Nov 02 14:18:21.705066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.704961    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"]
Nov 02 14:18:21.706075 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.706029    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:18:21.706150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.706106    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:18:21.706150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:21.706141    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:18:22.685374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:22.685335    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc" event={"ID":"32f3ee52-861f-4ea2-8a4d-2148fae634c2","Type":"ContainerStarted","Data":"13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d"}
Nov 02 14:18:22.685789 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:22.685401    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc" event={"ID":"32f3ee52-861f-4ea2-8a4d-2148fae634c2","Type":"ContainerStarted","Data":"bb674fbb68a8cf845928c337795a8d2e2f7ce7e06094a19796cdda09f826b3e5"}
Nov 02 14:18:22.746381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:22.746303    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc" podStartSLOduration=2.746287187 podStartE2EDuration="2.746287187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:18:22.745602182 +0000 UTC m=+2093.496906620" watchObservedRunningTime="2026-11-02 14:18:22.746287187 +0000 UTC m=+2093.497591625"
Nov 02 14:18:23.345028 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:23.344983    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:23.352851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:23.352824    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:23.700118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:23.700021    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:23.701111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:23.701082    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:25.172725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.172701    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:18:25.245341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.245308    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-podinfo\") pod \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") "
Nov 02 14:18:25.245525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.245501    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-envoy\") pod \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") "
Nov 02 14:18:25.245617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.245598    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-kube-api-access-vwsbj\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-kube-api-access-vwsbj\") pod \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") "
Nov 02 14:18:25.245684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.245644    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istiod-ca-cert\") pod \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") "
Nov 02 14:18:25.245684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.245677    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-certs\") pod \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") "
Nov 02 14:18:25.245835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.245710    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-socket\") pod \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") "
Nov 02 14:18:25.245911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.245834    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-credential-socket\") pod \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") "
Nov 02 14:18:25.245911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.245874    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-data\") pod \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") "
Nov 02 14:18:25.245911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.245900    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-token\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-token\") pod \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\" (UID: \"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa\") "
Nov 02 14:18:25.247595 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.247556    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-socket" pod "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" (UID: "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:25.247882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.247851    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-credential-socket" pod "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" (UID: "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:25.248008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.247908    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-data" pod "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" (UID: "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:25.248117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.248014    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-certs" pod "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" (UID: "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:25.248504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.248475    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istiod-ca-cert" pod "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" (UID: "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:18:25.249285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.249257    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-podinfo" pod "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" (UID: "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:18:25.249389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.249364    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-token" pod "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" (UID: "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:18:25.249389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.249388    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-kube-api-access-vwsbj" pod "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" (UID: "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa"). InnerVolumeSpecName "kube-api-access-vwsbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:18:25.250331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.250310    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-envoy" pod "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" (UID: "b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:25.352980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.352843    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:25.352980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.352920    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwsbj\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-kube-api-access-vwsbj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:25.353455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.353346    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:25.353455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.353398    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:25.353455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.353414    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:25.353455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.353427    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:25.353794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.353730    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:25.353794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.353752    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:25.353794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.353763    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:25.719754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.719658    5034 generic.go:356] "Generic (PLEG): container finished" podID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerID="869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0" exitCode=0
Nov 02 14:18:25.719754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.719703    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"
Nov 02 14:18:25.719754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.719737    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" event={"ID":"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa","Type":"ContainerDied","Data":"869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0"}
Nov 02 14:18:25.720058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.719856    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5" event={"ID":"b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa","Type":"ContainerDied","Data":"ed1b338236d5679668f3fb3a3fe662e2eba761e13821ae27a72fff39883340a4"}
Nov 02 14:18:25.720058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.719885    5034 scope.go:122] "RemoveContainer" containerID="869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0"
Nov 02 14:18:25.736436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.736418    5034 scope.go:122] "RemoveContainer" containerID="a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681"
Nov 02 14:18:25.747840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.747820    5034 scope.go:122] "RemoveContainer" containerID="869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0"
Nov 02 14:18:25.748174 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:18:25.748141    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0\": container with ID starting with 869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0 not found: ID does not exist" containerID="869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0"
Nov 02 14:18:25.748272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.748178    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0"} err="failed to get container status \"869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0\": rpc error: code = NotFound desc = could not find container \"869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0\": container with ID starting with 869a529783ad96cec1e9adf05ba7c6046b71f469d6529212b68765b9fed438b0 not found: ID does not exist"
Nov 02 14:18:25.748272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.748196    5034 scope.go:122] "RemoveContainer" containerID="a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681"
Nov 02 14:18:25.748469 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:18:25.748451    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681\": container with ID starting with a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681 not found: ID does not exist" containerID="a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681"
Nov 02 14:18:25.748522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.748477    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681"} err="failed to get container status \"a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681\": rpc error: code = NotFound desc = could not find container \"a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681\": container with ID starting with a52c05feb6a35d40e0db56f1b1607f52cb8fe7bad06ffb4138d6d988b78bd681 not found: ID does not exist"
Nov 02 14:18:25.778682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.778659    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"]
Nov 02 14:18:25.792219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:25.792198    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-5cbbfb-openshift-default-6c556b9b57-6p2w5"]
Nov 02 14:18:26.042297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:26.042266    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"]
Nov 02 14:18:26.048702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:26.048677    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175e7bfd-j29m9"]
Nov 02 14:18:27.397158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:27.397125    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" path="/var/lib/kubelet/pods/b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa/volumes"
Nov 02 14:18:27.397597 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:27.397582    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de80a299-3630-4777-a161-69b39ecae74a" path="/var/lib/kubelet/pods/de80a299-3630-4777-a161-69b39ecae74a/volumes"
Nov 02 14:18:30.358801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:30.358238    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:18:30.360605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:30.360582    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:18:33.030849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:33.030813    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"]
Nov 02 14:18:33.044159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:33.044128    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fad5039-nsbgh"]
Nov 02 14:18:33.397301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:33.397208    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e" path="/var/lib/kubelet/pods/fadf0a70-a2cd-4e60-b7c3-571ec2a44b6e/volumes"
Nov 02 14:18:35.031436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:35.031402    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"]
Nov 02 14:18:35.038234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:35.038210    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-abb03689-zljcv"]
Nov 02 14:18:35.399748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:35.399659    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30321d02-bf32-4cbe-aebb-54e11d957a8f" path="/var/lib/kubelet/pods/30321d02-bf32-4cbe-aebb-54e11d957a8f/volumes"
Nov 02 14:18:36.006398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:36.006354    5034 scope.go:122] "RemoveContainer" containerID="5964ca171127db3aecb9b96cc4cecfd097910ec44b7cf29ff5fb8eebf9ecc067"
Nov 02 14:18:36.017359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:36.017335    5034 scope.go:122] "RemoveContainer" containerID="c6fd394deabebabbf3930ee381f683264d92967f54673040b5342f9310e9cf13"
Nov 02 14:18:36.027551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:36.027518    5034 scope.go:122] "RemoveContainer" containerID="28ec3896d35f516cd134e7f9e8b94b49661801da494b2099f031524e25f42d82"
Nov 02 14:18:36.036943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:36.036903    5034 scope.go:122] "RemoveContainer" containerID="12f58e0e865b4888c0eff7b01d65e6c8f6a939bb881a769f82baadad8b1ba10b"
Nov 02 14:18:36.047525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:36.047507    5034 scope.go:122] "RemoveContainer" containerID="841bfc74187476ef1b6ec74ecc562f282a50b58df525d037a40730b2754f607e"
Nov 02 14:18:36.060035 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:36.060009    5034 scope.go:122] "RemoveContainer" containerID="cb9e9c0333465e222304b5a476d10e90f028743999ddf86740821269e8687d67"
Nov 02 14:18:36.069533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:36.069514    5034 scope.go:122] "RemoveContainer" containerID="7a804a2b2e0197615738518b496f195194310d446275fc099152adcd2b2ae8d2"
Nov 02 14:18:36.081499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:36.081465    5034 scope.go:122] "RemoveContainer" containerID="fc587a8519d3f25d71457987834daebf0e379382cf6d8036e760a74d7db19433"
Nov 02 14:18:38.031126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:38.031091    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"]
Nov 02 14:18:38.044266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:38.044238    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96abda6-4pnnm"]
Nov 02 14:18:39.396701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:39.396667    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733f02af-e007-4937-b4ef-02fa3c04ce0f" path="/var/lib/kubelet/pods/733f02af-e007-4937-b4ef-02fa3c04ce0f/volumes"
Nov 02 14:18:48.417902 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:18:48.417700    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:18:51.464359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:51.464321    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"]
Nov 02 14:18:51.465231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:51.465199    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc" podUID="32f3ee52-861f-4ea2-8a4d-2148fae634c2" containerName="istio-proxy" containerID="cri-o://13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d" gracePeriod=30
Nov 02 14:18:53.702426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:53.702386    5034 patch_prober.go:29] interesting pod/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 14:18:53.702901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:53.702453    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc" podUID="32f3ee52-861f-4ea2-8a4d-2148fae634c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:18:54.037757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:54.037718    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"]
Nov 02 14:18:54.048870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:54.048841    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d36463db-vvhlc"]
Nov 02 14:18:55.030484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:55.030452    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"]
Nov 02 14:18:55.043250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:55.043207    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-911e9272-s525t"]
Nov 02 14:18:55.397287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:55.397195    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348616d9-d3b9-438c-b932-57ebc5580d56" path="/var/lib/kubelet/pods/348616d9-d3b9-438c-b932-57ebc5580d56/volumes"
Nov 02 14:18:55.397812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:55.397769    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1352c3e-b748-4236-a22e-6c42cf097997" path="/var/lib/kubelet/pods/a1352c3e-b748-4236-a22e-6c42cf097997/volumes"
Nov 02 14:18:56.762060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.762036    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:56.856653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.856625    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-envoy\") pod \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") "
Nov 02 14:18:56.856839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.856694    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-credential-socket\") pod \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") "
Nov 02 14:18:56.856839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.856759    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-podinfo\") pod \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") "
Nov 02 14:18:56.856839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.856812    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-data\") pod \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") "
Nov 02 14:18:56.857045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.856849    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-token\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-token\") pod \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") "
Nov 02 14:18:56.857045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.856883    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-kube-api-access-v5x76\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-kube-api-access-v5x76\") pod \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") "
Nov 02 14:18:56.857045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.856913    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-certs\") pod \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") "
Nov 02 14:18:56.857045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.857010    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-socket\") pod \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") "
Nov 02 14:18:56.857270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.857050    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istiod-ca-cert\") pod \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\" (UID: \"32f3ee52-861f-4ea2-8a4d-2148fae634c2\") "
Nov 02 14:18:56.857674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.857624    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-data" pod "32f3ee52-861f-4ea2-8a4d-2148fae634c2" (UID: "32f3ee52-861f-4ea2-8a4d-2148fae634c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:56.857839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.857693    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-credential-socket" pod "32f3ee52-861f-4ea2-8a4d-2148fae634c2" (UID: "32f3ee52-861f-4ea2-8a4d-2148fae634c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:56.858220 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.858046    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istiod-ca-cert" pod "32f3ee52-861f-4ea2-8a4d-2148fae634c2" (UID: "32f3ee52-861f-4ea2-8a4d-2148fae634c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:18:56.858437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.858165    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-certs" pod "32f3ee52-861f-4ea2-8a4d-2148fae634c2" (UID: "32f3ee52-861f-4ea2-8a4d-2148fae634c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:56.858581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.858330    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-socket" pod "32f3ee52-861f-4ea2-8a4d-2148fae634c2" (UID: "32f3ee52-861f-4ea2-8a4d-2148fae634c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:56.860815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.860409    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-envoy" pod "32f3ee52-861f-4ea2-8a4d-2148fae634c2" (UID: "32f3ee52-861f-4ea2-8a4d-2148fae634c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:18:56.860815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.860764    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-token" pod "32f3ee52-861f-4ea2-8a4d-2148fae634c2" (UID: "32f3ee52-861f-4ea2-8a4d-2148fae634c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:18:56.861081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.861055    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-podinfo" pod "32f3ee52-861f-4ea2-8a4d-2148fae634c2" (UID: "32f3ee52-861f-4ea2-8a4d-2148fae634c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:18:56.861764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.861736    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-kube-api-access-v5x76" pod "32f3ee52-861f-4ea2-8a4d-2148fae634c2" (UID: "32f3ee52-861f-4ea2-8a4d-2148fae634c2"). InnerVolumeSpecName "kube-api-access-v5x76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:18:56.962327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.961467    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:56.962327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.961504    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:56.962327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.961530    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:56.962327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.961543    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:56.962327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.961556    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5x76\" (UniqueName: \"kubernetes.io/projected/32f3ee52-861f-4ea2-8a4d-2148fae634c2-kube-api-access-v5x76\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:56.962327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.961572    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:56.962327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.961591    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:56.962327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.961605    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:56.962327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:56.961618    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f3ee52-861f-4ea2-8a4d-2148fae634c2-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:18:57.023865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.022326    5034 generic.go:356] "Generic (PLEG): container finished" podID="32f3ee52-861f-4ea2-8a4d-2148fae634c2" containerID="13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d" exitCode=0
Nov 02 14:18:57.023865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.022554    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc" event={"ID":"32f3ee52-861f-4ea2-8a4d-2148fae634c2","Type":"ContainerDied","Data":"13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d"}
Nov 02 14:18:57.023865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.022637    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc" event={"ID":"32f3ee52-861f-4ea2-8a4d-2148fae634c2","Type":"ContainerDied","Data":"bb674fbb68a8cf845928c337795a8d2e2f7ce7e06094a19796cdda09f826b3e5"}
Nov 02 14:18:57.023865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.022948    5034 scope.go:122] "RemoveContainer" containerID="13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d"
Nov 02 14:18:57.023865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.022998    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"
Nov 02 14:18:57.036641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.036615    5034 scope.go:122] "RemoveContainer" containerID="13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d"
Nov 02 14:18:57.036932 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:18:57.036911    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d\": container with ID starting with 13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d not found: ID does not exist" containerID="13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d"
Nov 02 14:18:57.037068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.036943    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d"} err="failed to get container status \"13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d\": rpc error: code = NotFound desc = could not find container \"13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d\": container with ID starting with 13632fb2c065f836c4ec3e6da83414fe9739f2931fde3c1217073f1beb49ba1d not found: ID does not exist"
Nov 02 14:18:57.084402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.084375    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"]
Nov 02 14:18:57.090719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.090692    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-abc0f7-openshift-default-d6dc6bdc9-hrgxc"]
Nov 02 14:18:57.396932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:18:57.396855    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f3ee52-861f-4ea2-8a4d-2148fae634c2" path="/var/lib/kubelet/pods/32f3ee52-861f-4ea2-8a4d-2148fae634c2/volumes"
Nov 02 14:19:04.036156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:04.036122    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"]
Nov 02 14:19:04.052427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:04.052399    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d75cdf5a-vkfqq"]
Nov 02 14:19:05.420407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:05.420368    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc162e6c-c868-4755-8acb-1be07abc3cbc" path="/var/lib/kubelet/pods/bc162e6c-c868-4755-8acb-1be07abc3cbc/volumes"
Nov 02 14:19:10.518541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.518503    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"]
Nov 02 14:19:10.519723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.519699    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy"
Nov 02 14:19:10.519723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.519722    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy"
Nov 02 14:19:10.519962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.519763    5034 cpu_manager.go:414] "containerMap: removing container" podUID="32f3ee52-861f-4ea2-8a4d-2148fae634c2" containerName="istio-proxy"
Nov 02 14:19:10.519962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.519772    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f3ee52-861f-4ea2-8a4d-2148fae634c2" containerName="istio-proxy"
Nov 02 14:19:10.520071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.520031    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy"
Nov 02 14:19:10.520071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.520049    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f3ee52-861f-4ea2-8a4d-2148fae634c2" containerName="istio-proxy"
Nov 02 14:19:10.524261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.524241    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.544375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.544346    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"]
Nov 02 14:19:10.551229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.551205    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:19:10.551355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.551300    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:19:10.551457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.551441    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:19:10.553680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.553660    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:19:10.620647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.620613    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27lw\" (UniqueName: \"kubernetes.io/projected/c93f5d70-a180-4d84-a54d-7093e773b008-kube-api-access-j27lw\") pod \"maas-api-revoke-sub-a8417d92-ht4rq\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.620963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.620929    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8417d92-ht4rq\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.621132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.620997    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8417d92-ht4rq\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.722307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.722275    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8417d92-ht4rq\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.722524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.722329    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8417d92-ht4rq\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.722524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.722435    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27lw\" (UniqueName: \"kubernetes.io/projected/c93f5d70-a180-4d84-a54d-7093e773b008-kube-api-access-j27lw\") pod \"maas-api-revoke-sub-a8417d92-ht4rq\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.723051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.723024    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8417d92-ht4rq\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.724818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.724796    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8417d92-ht4rq\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.749529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.749497    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27lw\" (UniqueName: \"kubernetes.io/projected/c93f5d70-a180-4d84-a54d-7093e773b008-kube-api-access-j27lw\") pod \"maas-api-revoke-sub-a8417d92-ht4rq\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:10.840172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:10.840096    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:11.149999 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:19:11.149954    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93f5d70_a180_4d84_a54d_7093e773b008.slice/crio-d467f293af58be1e7207d3ef465ee608dd83b55fb76947297b3d1c9c0322f598 WatchSource:0}: Error finding container d467f293af58be1e7207d3ef465ee608dd83b55fb76947297b3d1c9c0322f598: Status 404 returned error can't find the container with id d467f293af58be1e7207d3ef465ee608dd83b55fb76947297b3d1c9c0322f598
Nov 02 14:19:11.150636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:11.150609    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"]
Nov 02 14:19:11.181568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:11.181533    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq" event={"ID":"c93f5d70-a180-4d84-a54d-7093e773b008","Type":"ContainerStarted","Data":"d467f293af58be1e7207d3ef465ee608dd83b55fb76947297b3d1c9c0322f598"}
Nov 02 14:19:12.194003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:12.193966    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq" event={"ID":"c93f5d70-a180-4d84-a54d-7093e773b008","Type":"ContainerStarted","Data":"87056c376d77fb7a003b98b6da2ecf6d90bf463ad7477912ca9d8e973b424139"}
Nov 02 14:19:12.240658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:12.240607    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq" podStartSLOduration=2.240593145 podStartE2EDuration="2.240593145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:12.238008168 +0000 UTC m=+2142.989312605" watchObservedRunningTime="2026-11-02 14:19:12.240593145 +0000 UTC m=+2142.991897586"
Nov 02 14:19:13.204918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:13.204882    5034 generic.go:356] "Generic (PLEG): container finished" podID="c93f5d70-a180-4d84-a54d-7093e773b008" containerID="87056c376d77fb7a003b98b6da2ecf6d90bf463ad7477912ca9d8e973b424139" exitCode=0
Nov 02 14:19:13.205498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:13.204971    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq" event={"ID":"c93f5d70-a180-4d84-a54d-7093e773b008","Type":"ContainerDied","Data":"87056c376d77fb7a003b98b6da2ecf6d90bf463ad7477912ca9d8e973b424139"}
Nov 02 14:19:14.403205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.403181    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:14.491354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.491321    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-openshift-ca\") pod \"c93f5d70-a180-4d84-a54d-7093e773b008\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") "
Nov 02 14:19:14.491561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.491522    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93f5d70-a180-4d84-a54d-7093e773b008-kube-api-access-j27lw\" (UniqueName: \"kubernetes.io/projected/c93f5d70-a180-4d84-a54d-7093e773b008-kube-api-access-j27lw\") pod \"c93f5d70-a180-4d84-a54d-7093e773b008\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") "
Nov 02 14:19:14.491656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.491611    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-xks-ca\") pod \"c93f5d70-a180-4d84-a54d-7093e773b008\" (UID: \"c93f5d70-a180-4d84-a54d-7093e773b008\") "
Nov 02 14:19:14.491820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.491754    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-openshift-ca" pod "c93f5d70-a180-4d84-a54d-7093e773b008" (UID: "c93f5d70-a180-4d84-a54d-7093e773b008"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:14.492593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.492568    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:14.494180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.494153    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93f5d70-a180-4d84-a54d-7093e773b008-kube-api-access-j27lw" pod "c93f5d70-a180-4d84-a54d-7093e773b008" (UID: "c93f5d70-a180-4d84-a54d-7093e773b008"). InnerVolumeSpecName "kube-api-access-j27lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:14.494427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.494401    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-xks-ca" pod "c93f5d70-a180-4d84-a54d-7093e773b008" (UID: "c93f5d70-a180-4d84-a54d-7093e773b008"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:19:14.594289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.594258    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j27lw\" (UniqueName: \"kubernetes.io/projected/c93f5d70-a180-4d84-a54d-7093e773b008-kube-api-access-j27lw\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:14.594289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:14.594285    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f5d70-a180-4d84-a54d-7093e773b008-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:15.225749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:15.225720    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"
Nov 02 14:19:15.225995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:15.225753    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq" event={"ID":"c93f5d70-a180-4d84-a54d-7093e773b008","Type":"ContainerDied","Data":"d467f293af58be1e7207d3ef465ee608dd83b55fb76947297b3d1c9c0322f598"}
Nov 02 14:19:15.225995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:15.225853    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d467f293af58be1e7207d3ef465ee608dd83b55fb76947297b3d1c9c0322f598"
Nov 02 14:19:18.415218 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:18.415170    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:19:20.858275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.858240    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"]
Nov 02 14:19:20.858997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.858980    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c93f5d70-a180-4d84-a54d-7093e773b008" containerName="revoke-keys"
Nov 02 14:19:20.858997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.858998    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93f5d70-a180-4d84-a54d-7093e773b008" containerName="revoke-keys"
Nov 02 14:19:20.859113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.859060    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy"
Nov 02 14:19:20.859113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.859065    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy"
Nov 02 14:19:20.859178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.859170    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a55e6c-c9fb-40f0-a9c5-e8e9fd057dfa" containerName="istio-proxy"
Nov 02 14:19:20.859214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.859183    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93f5d70-a180-4d84-a54d-7093e773b008" containerName="revoke-keys"
Nov 02 14:19:20.862977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.862959    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:20.883041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.883018    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"]
Nov 02 14:19:20.887480 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.887455    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb9aa177-dpdld\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:20.887692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.887669    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d429w\" (UniqueName: \"kubernetes.io/projected/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-kube-api-access-d429w\") pod \"maas-api-revoke-sub-eb9aa177-dpdld\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:20.887841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.887774    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb9aa177-dpdld\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:20.902832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.891328    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:19:20.902832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.891506    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:19:20.902832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.892276    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:19:20.902832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.892489    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:19:20.989605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.989572    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb9aa177-dpdld\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:20.989810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.989793    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb9aa177-dpdld\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:20.990002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.989982    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d429w\" (UniqueName: \"kubernetes.io/projected/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-kube-api-access-d429w\") pod \"maas-api-revoke-sub-eb9aa177-dpdld\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:20.990391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.990364    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb9aa177-dpdld\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:20.992693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:20.992671    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb9aa177-dpdld\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:21.021909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:21.021890    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d429w\" (UniqueName: \"kubernetes.io/projected/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-kube-api-access-d429w\") pod \"maas-api-revoke-sub-eb9aa177-dpdld\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:21.048665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:21.048641    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"]
Nov 02 14:19:21.053419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:21.053401    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-547482f3-jbrgv"]
Nov 02 14:19:21.177219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:21.177152    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:21.401663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:21.401630    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89ab83e-c555-4652-9cc4-9d43af3bb0c3" path="/var/lib/kubelet/pods/f89ab83e-c555-4652-9cc4-9d43af3bb0c3/volumes"
Nov 02 14:19:21.488950 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:19:21.488918    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ed2bf9_9f85_4731_bfd4_c1f7503caa9c.slice/crio-dd59557cb1afa037da384ee067db1286fac3d55258827a1fbe934edc831cbb5e WatchSource:0}: Error finding container dd59557cb1afa037da384ee067db1286fac3d55258827a1fbe934edc831cbb5e: Status 404 returned error can't find the container with id dd59557cb1afa037da384ee067db1286fac3d55258827a1fbe934edc831cbb5e
Nov 02 14:19:21.490169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:21.490140    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"]
Nov 02 14:19:22.286613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:22.286579    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld" event={"ID":"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c","Type":"ContainerStarted","Data":"20913736827a4d19a125c7d5d85ae5821199e3b8c7e75acb028e1d52cbb1fc4c"}
Nov 02 14:19:22.287067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:22.286644    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld" event={"ID":"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c","Type":"ContainerStarted","Data":"dd59557cb1afa037da384ee067db1286fac3d55258827a1fbe934edc831cbb5e"}
Nov 02 14:19:22.331661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:22.331609    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld" podStartSLOduration=2.331593926 podStartE2EDuration="2.331593926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:22.330398087 +0000 UTC m=+2153.081702525" watchObservedRunningTime="2026-11-02 14:19:22.331593926 +0000 UTC m=+2153.082898386"
Nov 02 14:19:23.312451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:23.312412    5034 generic.go:356] "Generic (PLEG): container finished" podID="a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c" containerID="20913736827a4d19a125c7d5d85ae5821199e3b8c7e75acb028e1d52cbb1fc4c" exitCode=0
Nov 02 14:19:23.312981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:23.312495    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld" event={"ID":"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c","Type":"ContainerDied","Data":"20913736827a4d19a125c7d5d85ae5821199e3b8c7e75acb028e1d52cbb1fc4c"}
Nov 02 14:19:24.509057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.509029    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:24.544615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.544589    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-xks-ca\") pod \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") "
Nov 02 14:19:24.544813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.544641    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-openshift-ca\") pod \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") "
Nov 02 14:19:24.544813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.544753    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-kube-api-access-d429w\" (UniqueName: \"kubernetes.io/projected/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-kube-api-access-d429w\") pod \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\" (UID: \"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c\") "
Nov 02 14:19:24.545120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.545093    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-openshift-ca" pod "a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c" (UID: "a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:24.545507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.545490    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:24.546816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.546773    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-xks-ca" pod "a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c" (UID: "a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:19:24.546922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.546858    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-kube-api-access-d429w" pod "a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c" (UID: "a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c"). InnerVolumeSpecName "kube-api-access-d429w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:24.647239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.647161    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d429w\" (UniqueName: \"kubernetes.io/projected/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-kube-api-access-d429w\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:24.647239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:24.647195    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:25.330413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:25.330377    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld" event={"ID":"a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c","Type":"ContainerDied","Data":"dd59557cb1afa037da384ee067db1286fac3d55258827a1fbe934edc831cbb5e"}
Nov 02 14:19:25.330413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:25.330407    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"
Nov 02 14:19:25.330640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:25.330437    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd59557cb1afa037da384ee067db1286fac3d55258827a1fbe934edc831cbb5e"
Nov 02 14:19:31.888133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:31.888100    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"]
Nov 02 14:19:31.888567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:31.888366    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy" containerID="cri-o://fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3" gracePeriod=30
Nov 02 14:19:32.686453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:32.686412    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"]
Nov 02 14:19:32.686657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:32.686624    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy" containerID="cri-o://68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c" gracePeriod=30
Nov 02 14:19:32.700338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:32.700305    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"]
Nov 02 14:19:32.700629 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:32.700600    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy" containerID="cri-o://1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5" gracePeriod=30
Nov 02 14:19:32.704855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:32.704827    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"]
Nov 02 14:19:32.705069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:32.705045    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy" containerID="cri-o://bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273" gracePeriod=30
Nov 02 14:19:34.834270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:34.834236    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7036-e568de"]
Nov 02 14:19:34.835064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:34.835042    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c" containerName="revoke-keys"
Nov 02 14:19:34.835064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:34.835066    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c" containerName="revoke-keys"
Nov 02 14:19:34.835349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:34.835331    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c" containerName="revoke-keys"
Nov 02 14:19:34.839142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:34.839122    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:34.855654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:34.855630    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7jzh4\""
Nov 02 14:19:34.858817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:34.857876    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7036-e568de"]
Nov 02 14:19:34.937039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:34.937007    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h5gn\" (UniqueName: \"kubernetes.io/projected/6f10d14d-fb6a-49fd-9b04-0f616f192270-kube-api-access-2h5gn\") pod \"test-curl-7036-e568de\" (UID: \"6f10d14d-fb6a-49fd-9b04-0f616f192270\") " pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:35.039250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:35.039216    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h5gn\" (UniqueName: \"kubernetes.io/projected/6f10d14d-fb6a-49fd-9b04-0f616f192270-kube-api-access-2h5gn\") pod \"test-curl-7036-e568de\" (UID: \"6f10d14d-fb6a-49fd-9b04-0f616f192270\") " pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:35.064052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:35.064017    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h5gn\" (UniqueName: \"kubernetes.io/projected/6f10d14d-fb6a-49fd-9b04-0f616f192270-kube-api-access-2h5gn\") pod \"test-curl-7036-e568de\" (UID: \"6f10d14d-fb6a-49fd-9b04-0f616f192270\") " pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:35.153071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:35.152983    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:35.500827 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:19:35.500743    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f10d14d_fb6a_49fd_9b04_0f616f192270.slice/crio-585ae24dfe9dda06dffa05f4422284717e4a8611dbb95634ea196f383fd93120 WatchSource:0}: Error finding container 585ae24dfe9dda06dffa05f4422284717e4a8611dbb95634ea196f383fd93120: Status 404 returned error can't find the container with id 585ae24dfe9dda06dffa05f4422284717e4a8611dbb95634ea196f383fd93120
Nov 02 14:19:35.503047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:35.503019    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7036-e568de"]
Nov 02 14:19:36.170024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.169984    5034 scope.go:122] "RemoveContainer" containerID="e13fd2ba43c9ab330b5ff9c774b4c398665cc1888b20b88e879b832fd80c6837"
Nov 02 14:19:36.185851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.185825    5034 scope.go:122] "RemoveContainer" containerID="3756e4da9c6d5dbae2eafb8608c38cad2757208aa174d09141b76bb0fd5b74cc"
Nov 02 14:19:36.197752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.197728    5034 scope.go:122] "RemoveContainer" containerID="7b1446d033186d2da92c92b86608103d9206a54a2932b1e0af6ea656db70a24a"
Nov 02 14:19:36.207907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.207889    5034 scope.go:122] "RemoveContainer" containerID="3edbbe8920abadca619a701b9bb7ed07cca2441305528bce8a691bcf66da027e"
Nov 02 14:19:36.219819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.219771    5034 scope.go:122] "RemoveContainer" containerID="7c9881370d161fe85f9c186a3d384d77e79ce276d6b8778825a2c5b2f7248ec5"
Nov 02 14:19:36.228848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.228827    5034 scope.go:122] "RemoveContainer" containerID="8e5469121ea12d48d4c6b73e63c12e9dddf1a51ccf7d6f60953c9ca0276bc061"
Nov 02 14:19:36.238243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.238218    5034 scope.go:122] "RemoveContainer" containerID="6c57bfc996708e01bfb1c2841f2dd53ce462344a5ba533e82ac7c127fc952760"
Nov 02 14:19:36.247929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.247911    5034 scope.go:122] "RemoveContainer" containerID="7f7296b27507ddb1268b50aeab436e67d5a3e48918a456d781d45889739c9192"
Nov 02 14:19:36.257440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.257420    5034 scope.go:122] "RemoveContainer" containerID="307343b76c1ec17c953e8e4c3d6701fb915aa1e65e50e5525e0e988a48d0fd99"
Nov 02 14:19:36.266518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.266497    5034 scope.go:122] "RemoveContainer" containerID="50def01b2cd6f022292b5df7c8b02739b8a7e48e9955cf912b996f6396509539"
Nov 02 14:19:36.275944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.275924    5034 scope.go:122] "RemoveContainer" containerID="d257e02e4eee521852630f1c68e083e51519235d6236df45d605027d614a3545"
Nov 02 14:19:36.471047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.470961    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7036-e568de" event={"ID":"6f10d14d-fb6a-49fd-9b04-0f616f192270","Type":"ContainerStarted","Data":"c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835"}
Nov 02 14:19:36.471047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.471015    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7036-e568de" event={"ID":"6f10d14d-fb6a-49fd-9b04-0f616f192270","Type":"ContainerStarted","Data":"585ae24dfe9dda06dffa05f4422284717e4a8611dbb95634ea196f383fd93120"}
Nov 02 14:19:36.512258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:36.512041    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7036-e568de" podStartSLOduration=2.512022643 podStartE2EDuration="2.512022643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:36.511625606 +0000 UTC m=+2167.262930047" watchObservedRunningTime="2026-11-02 14:19:36.512022643 +0000 UTC m=+2167.263327083"
Nov 02 14:19:37.182341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.182318    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 14:19:37.274254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274223    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-socket\") pod \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") "
Nov 02 14:19:37.274442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274263    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-certs\") pod \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") "
Nov 02 14:19:37.274442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274312    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istiod-ca-cert\") pod \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") "
Nov 02 14:19:37.274571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274445    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-data\") pod \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") "
Nov 02 14:19:37.274743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274728    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-token\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-token\") pod \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") "
Nov 02 14:19:37.274862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274533    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-socket" pod "856cd9ab-9a6f-4ba3-b8ea-87b83d868832" (UID: "856cd9ab-9a6f-4ba3-b8ea-87b83d868832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:37.274862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274639    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-certs" pod "856cd9ab-9a6f-4ba3-b8ea-87b83d868832" (UID: "856cd9ab-9a6f-4ba3-b8ea-87b83d868832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:37.274862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274667    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-data" pod "856cd9ab-9a6f-4ba3-b8ea-87b83d868832" (UID: "856cd9ab-9a6f-4ba3-b8ea-87b83d868832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:37.274862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274827    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-podinfo\") pod \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") "
Nov 02 14:19:37.274862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274830    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istiod-ca-cert" pod "856cd9ab-9a6f-4ba3-b8ea-87b83d868832" (UID: "856cd9ab-9a6f-4ba3-b8ea-87b83d868832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:37.274862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274856    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-envoy\") pod \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") "
Nov 02 14:19:37.275218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274895    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-credential-socket\") pod \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") "
Nov 02 14:19:37.275218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.274961    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-kube-api-access-nzsz9\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-kube-api-access-nzsz9\") pod \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\" (UID: \"856cd9ab-9a6f-4ba3-b8ea-87b83d868832\") "
Nov 02 14:19:37.275425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.275400    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-credential-socket" pod "856cd9ab-9a6f-4ba3-b8ea-87b83d868832" (UID: "856cd9ab-9a6f-4ba3-b8ea-87b83d868832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:37.276240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.276214    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:37.276364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.276242    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:37.276364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.276257    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:37.276364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.276275    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:37.276364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.276289    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:37.277460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.277431    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-podinfo" pod "856cd9ab-9a6f-4ba3-b8ea-87b83d868832" (UID: "856cd9ab-9a6f-4ba3-b8ea-87b83d868832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:19:37.277673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.277648    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-envoy" pod "856cd9ab-9a6f-4ba3-b8ea-87b83d868832" (UID: "856cd9ab-9a6f-4ba3-b8ea-87b83d868832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:37.277934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.277905    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-token" pod "856cd9ab-9a6f-4ba3-b8ea-87b83d868832" (UID: "856cd9ab-9a6f-4ba3-b8ea-87b83d868832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:37.278282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.278262    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-kube-api-access-nzsz9" pod "856cd9ab-9a6f-4ba3-b8ea-87b83d868832" (UID: "856cd9ab-9a6f-4ba3-b8ea-87b83d868832"). InnerVolumeSpecName "kube-api-access-nzsz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:37.378315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.378283    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzsz9\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-kube-api-access-nzsz9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:37.378315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.378311    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:37.378315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.378320    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:37.378556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.378328    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856cd9ab-9a6f-4ba3-b8ea-87b83d868832-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:37.399811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.399748    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7036-e568de"]
Nov 02 14:19:37.401170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.401141    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7036-e568de"]
Nov 02 14:19:37.483199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.483162    5034 generic.go:356] "Generic (PLEG): container finished" podID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerID="fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3" exitCode=0
Nov 02 14:19:37.483370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.483227    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"
Nov 02 14:19:37.483370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.483259    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" event={"ID":"856cd9ab-9a6f-4ba3-b8ea-87b83d868832","Type":"ContainerDied","Data":"fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3"}
Nov 02 14:19:37.483370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.483346    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd" event={"ID":"856cd9ab-9a6f-4ba3-b8ea-87b83d868832","Type":"ContainerDied","Data":"e5766821b0a005debbbae0b51f103f7bec0abfc975d678a374ca61d5fe685644"}
Nov 02 14:19:37.483550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.483398    5034 scope.go:122] "RemoveContainer" containerID="fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3"
Nov 02 14:19:37.494476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.494456    5034 scope.go:122] "RemoveContainer" containerID="357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5"
Nov 02 14:19:37.504383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.504359    5034 scope.go:122] "RemoveContainer" containerID="fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3"
Nov 02 14:19:37.504718 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:37.504690    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3\": container with ID starting with fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3 not found: ID does not exist" containerID="fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3"
Nov 02 14:19:37.504853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.504727    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3"} err="failed to get container status \"fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3\": rpc error: code = NotFound desc = could not find container \"fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3\": container with ID starting with fa595c32b6867dd2742eef45a652c81ea5919944ca8a49143b3e6b58325bb7f3 not found: ID does not exist"
Nov 02 14:19:37.504853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.504753    5034 scope.go:122] "RemoveContainer" containerID="357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5"
Nov 02 14:19:37.505126 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:37.505096    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5\": container with ID starting with 357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5 not found: ID does not exist" containerID="357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5"
Nov 02 14:19:37.505259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.505131    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5"} err="failed to get container status \"357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5\": rpc error: code = NotFound desc = could not find container \"357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5\": container with ID starting with 357c1d1ac1195718c35e19b6733fbadf856415ebca8b5cbfa4f2ec62fea39fb5 not found: ID does not exist"
Nov 02 14:19:37.541806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.541769    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"]
Nov 02 14:19:37.545581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:37.545560    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-wdsbd"]
Nov 02 14:19:38.116188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.116158    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:19:38.118291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.118268    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:19:38.165352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.165330    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:19:38.203153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203126    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-certs\") pod \"72df336e-41ed-45c2-b132-7c715685584e\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") "
Nov 02 14:19:38.203554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203208    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-credential-socket\") pod \"72df336e-41ed-45c2-b132-7c715685584e\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") "
Nov 02 14:19:38.203554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203262    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-kube-api-access-vfqvx\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-kube-api-access-vfqvx\") pod \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") "
Nov 02 14:19:38.203554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203314    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-data\") pod \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") "
Nov 02 14:19:38.203554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203425    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-certs" pod "72df336e-41ed-45c2-b132-7c715685584e" (UID: "72df336e-41ed-45c2-b132-7c715685584e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.203554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203531    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-credential-socket" pod "72df336e-41ed-45c2-b132-7c715685584e" (UID: "72df336e-41ed-45c2-b132-7c715685584e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.203751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203658    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-data" pod "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" (UID: "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.203805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203757    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-envoy\") pod \"72df336e-41ed-45c2-b132-7c715685584e\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") "
Nov 02 14:19:38.203856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203840    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-envoy\") pod \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") "
Nov 02 14:19:38.203907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203893    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-token\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-token\") pod \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") "
Nov 02 14:19:38.203958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203928    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-token\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-token\") pod \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") "
Nov 02 14:19:38.204017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203956    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-socket\") pod \"72df336e-41ed-45c2-b132-7c715685584e\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") "
Nov 02 14:19:38.204017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.203990    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-certs\") pod \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") "
Nov 02 14:19:38.204131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204021    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-envoy\") pod \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") "
Nov 02 14:19:38.204131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204054    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istiod-ca-cert\") pod \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") "
Nov 02 14:19:38.204131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204083    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-credential-socket\") pod \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") "
Nov 02 14:19:38.204131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204128    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-socket\") pod \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") "
Nov 02 14:19:38.204355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204194    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-certs\") pod \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") "
Nov 02 14:19:38.204355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204253    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-podinfo\") pod \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") "
Nov 02 14:19:38.204355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204263    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-socket" pod "72df336e-41ed-45c2-b132-7c715685584e" (UID: "72df336e-41ed-45c2-b132-7c715685584e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.204355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204303    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-kube-api-access-r2n68\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-kube-api-access-r2n68\") pod \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") "
Nov 02 14:19:38.204581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204374    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istiod-ca-cert\") pod \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") "
Nov 02 14:19:38.204581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204461    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-kube-api-access-lm5fj\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-kube-api-access-lm5fj\") pod \"72df336e-41ed-45c2-b132-7c715685584e\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") "
Nov 02 14:19:38.204581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204532    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-credential-socket" pod "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" (UID: "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.204844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204823    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istiod-ca-cert" pod "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" (UID: "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:38.204936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204862    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-credential-socket\") pod \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") "
Nov 02 14:19:38.204936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204896    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-podinfo\") pod \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") "
Nov 02 14:19:38.204936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204932    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-data\") pod \"72df336e-41ed-45c2-b132-7c715685584e\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") "
Nov 02 14:19:38.205174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.204962    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72df336e-41ed-45c2-b132-7c715685584e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72df336e-41ed-45c2-b132-7c715685584e-istio-podinfo\") pod \"72df336e-41ed-45c2-b132-7c715685584e\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") "
Nov 02 14:19:38.205242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.205181    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-istio-token\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-istio-token\") pod \"72df336e-41ed-45c2-b132-7c715685584e\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") "
Nov 02 14:19:38.205242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.205227    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-data\") pod \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\" (UID: \"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c\") "
Nov 02 14:19:38.205352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.205266    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-socket\") pod \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\" (UID: \"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb\") "
Nov 02 14:19:38.205352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.205304    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72df336e-41ed-45c2-b132-7c715685584e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72df336e-41ed-45c2-b132-7c715685584e-istiod-ca-cert\") pod \"72df336e-41ed-45c2-b132-7c715685584e\" (UID: \"72df336e-41ed-45c2-b132-7c715685584e\") "
Nov 02 14:19:38.206793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206294    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.206793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206319    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.206793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206334    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.206793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206350    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.206793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206364    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.206793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206378    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.206793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206419    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-kube-api-access-vfqvx" pod "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" (UID: "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c"). InnerVolumeSpecName "kube-api-access-vfqvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:38.206793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206663    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-certs" pod "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" (UID: "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.207312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206766    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-token" pod "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" (UID: "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:38.207312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206821    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-envoy" pod "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" (UID: "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.207312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206826    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72df336e-41ed-45c2-b132-7c715685584e-istiod-ca-cert" pod "72df336e-41ed-45c2-b132-7c715685584e" (UID: "72df336e-41ed-45c2-b132-7c715685584e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:38.207312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.206904    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-kube-api-access-lm5fj" pod "72df336e-41ed-45c2-b132-7c715685584e" (UID: "72df336e-41ed-45c2-b132-7c715685584e"). InnerVolumeSpecName "kube-api-access-lm5fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:38.207312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.207103    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-socket" pod "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" (UID: "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.207312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.207300    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-certs" pod "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" (UID: "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.207649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.207591    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istiod-ca-cert" pod "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" (UID: "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:19:38.208657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.208446    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-envoy" pod "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" (UID: "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.208887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.208859    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-data" pod "72df336e-41ed-45c2-b132-7c715685584e" (UID: "72df336e-41ed-45c2-b132-7c715685584e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.209074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.209056    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-credential-socket" pod "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" (UID: "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.209252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.209232    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-data" pod "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" (UID: "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.209462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.209437    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-socket" pod "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" (UID: "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.209990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.209958    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-kube-api-access-r2n68" pod "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" (UID: "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb"). InnerVolumeSpecName "kube-api-access-r2n68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:38.210238 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.210152    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72df336e-41ed-45c2-b132-7c715685584e-istio-podinfo" pod "72df336e-41ed-45c2-b132-7c715685584e" (UID: "72df336e-41ed-45c2-b132-7c715685584e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:19:38.210238 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.210155    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-podinfo" pod "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" (UID: "9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:19:38.210238 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.210212    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-istio-token" pod "72df336e-41ed-45c2-b132-7c715685584e" (UID: "72df336e-41ed-45c2-b132-7c715685584e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:38.210399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.210377    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-podinfo" pod "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" (UID: "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:19:38.211848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.211770    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-envoy" pod "72df336e-41ed-45c2-b132-7c715685584e" (UID: "72df336e-41ed-45c2-b132-7c715685584e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:19:38.211948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.211923    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-token" pod "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" (UID: "2d1ff927-cad5-4b00-b3ae-39d9de8ce94c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:38.307622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307550    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72df336e-41ed-45c2-b132-7c715685584e-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.307622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307581    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.307622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307590    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.307622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307598    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.307622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307605    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72df336e-41ed-45c2-b132-7c715685584e-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.307622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307614    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfqvx\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-kube-api-access-vfqvx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.307622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307623    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.307622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307632    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307644    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307655    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307662    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307670    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307678    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307685    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307693    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307702    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2n68\" (UniqueName: \"kubernetes.io/projected/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-kube-api-access-r2n68\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307712    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307724    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm5fj\" (UniqueName: \"kubernetes.io/projected/72df336e-41ed-45c2-b132-7c715685584e-kube-api-access-lm5fj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307732    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307740    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.308062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.307749    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72df336e-41ed-45c2-b132-7c715685584e-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:38.492908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.492872    5034 generic.go:356] "Generic (PLEG): container finished" podID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerID="68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c" exitCode=0
Nov 02 14:19:38.493093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.492946    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"
Nov 02 14:19:38.493093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.492966    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" event={"ID":"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c","Type":"ContainerDied","Data":"68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c"}
Nov 02 14:19:38.493093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.493025    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn" event={"ID":"2d1ff927-cad5-4b00-b3ae-39d9de8ce94c","Type":"ContainerDied","Data":"a9292e72d948b30353ad947ee7ad23ba86dca335742244282e585faf866fc7f9"}
Nov 02 14:19:38.493093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.493048    5034 scope.go:122] "RemoveContainer" containerID="68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c"
Nov 02 14:19:38.496111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.496083    5034 generic.go:356] "Generic (PLEG): container finished" podID="72df336e-41ed-45c2-b132-7c715685584e" containerID="bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273" exitCode=0
Nov 02 14:19:38.496255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.496137    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"
Nov 02 14:19:38.496255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.496161    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" event={"ID":"72df336e-41ed-45c2-b132-7c715685584e","Type":"ContainerDied","Data":"bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273"}
Nov 02 14:19:38.496255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.496207    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb" event={"ID":"72df336e-41ed-45c2-b132-7c715685584e","Type":"ContainerDied","Data":"bc6b19dd90d21d5d5f4e37a44496e2e7d44917001592c43faa34648252e72bdc"}
Nov 02 14:19:38.498048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.498023    5034 generic.go:356] "Generic (PLEG): container finished" podID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerID="1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5" exitCode=0
Nov 02 14:19:38.498187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.498084    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"
Nov 02 14:19:38.498187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.498119    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" event={"ID":"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb","Type":"ContainerDied","Data":"1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5"}
Nov 02 14:19:38.498187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.498161    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w" event={"ID":"9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb","Type":"ContainerDied","Data":"9434c766463df280499193490969ca8d186f4aa8503e94596bc4fc49e686a935"}
Nov 02 14:19:38.498368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.498225    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7036-e568de" podUID="6f10d14d-fb6a-49fd-9b04-0f616f192270" containerName="test-curl-7036-e568de" containerID="cri-o://c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835" gracePeriod=2
Nov 02 14:19:38.512691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.512670    5034 scope.go:122] "RemoveContainer" containerID="68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c"
Nov 02 14:19:38.513001 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:38.512978    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c\": container with ID starting with 68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c not found: ID does not exist" containerID="68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c"
Nov 02 14:19:38.513095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.513006    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c"} err="failed to get container status \"68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c\": rpc error: code = NotFound desc = could not find container \"68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c\": container with ID starting with 68c51836d744eef89d8adc3d982bc2e8ccf4d316bf963444aadc7706c7ce766c not found: ID does not exist"
Nov 02 14:19:38.513095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.513023    5034 scope.go:122] "RemoveContainer" containerID="bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273"
Nov 02 14:19:38.550838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.547493    5034 scope.go:122] "RemoveContainer" containerID="bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273"
Nov 02 14:19:38.570868 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:38.567957    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273\": container with ID starting with bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273 not found: ID does not exist" containerID="bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273"
Nov 02 14:19:38.570868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.568004    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273"} err="failed to get container status \"bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273\": rpc error: code = NotFound desc = could not find container \"bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273\": container with ID starting with bb808c47d267017356e8c22e90e0fe19f3bec887d2f9359a3fa3a00668b41273 not found: ID does not exist"
Nov 02 14:19:38.570868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.568031    5034 scope.go:122] "RemoveContainer" containerID="1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5"
Nov 02 14:19:38.627430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.627313    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"]
Nov 02 14:19:38.732670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.731368    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-kc6dn"]
Nov 02 14:19:38.744227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.743103    5034 scope.go:122] "RemoveContainer" containerID="1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5"
Nov 02 14:19:38.744561 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:38.744459    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5\": container with ID starting with 1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5 not found: ID does not exist" containerID="1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5"
Nov 02 14:19:38.744561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.744501    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5"} err="failed to get container status \"1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5\": rpc error: code = NotFound desc = could not find container \"1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5\": container with ID starting with 1e6f2c057bfe4985ae31e73ccffb5da05068b1dc61f5bb2c5234114eeac973e5 not found: ID does not exist"
Nov 02 14:19:38.784631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.784590    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"]
Nov 02 14:19:38.785697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785668    5034 cpu_manager.go:414] "containerMap: removing container" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy"
Nov 02 14:19:38.785697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785695    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785718    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785726    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785745    5034 cpu_manager.go:414] "containerMap: removing container" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785753    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785801    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785809    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785827    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6f10d14d-fb6a-49fd-9b04-0f616f192270" containerName="test-curl-7036-e568de"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785833    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f10d14d-fb6a-49fd-9b04-0f616f192270" containerName="test-curl-7036-e568de"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785857    5034 cpu_manager.go:414] "containerMap: removing container" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy"
Nov 02 14:19:38.785886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.785864    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy"
Nov 02 14:19:38.786217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.786030    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" containerName="istio-proxy"
Nov 02 14:19:38.786217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.786043    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f10d14d-fb6a-49fd-9b04-0f616f192270" containerName="test-curl-7036-e568de"
Nov 02 14:19:38.786217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.786058    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="72df336e-41ed-45c2-b132-7c715685584e" containerName="istio-proxy"
Nov 02 14:19:38.786217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.786075    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy"
Nov 02 14:19:38.786217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.786084    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" containerName="istio-proxy"
Nov 02 14:19:38.795128 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:38.795090    5034 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7036-e568de\" is forbidden: User \"system:node:ip-10-0-139-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-26.ec2.internal' and this object" podUID="6f10d14d-fb6a-49fd-9b04-0f616f192270" pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:38.804397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.804366    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"]
Nov 02 14:19:38.804397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.804389    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:38.804596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.804417    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lnjkb"]
Nov 02 14:19:38.804596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.804445    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"]
Nov 02 14:19:38.805958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.805938    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"]
Nov 02 14:19:38.809884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.809866    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-lfl6w"]
Nov 02 14:19:38.819448 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:38.819411    5034 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7036-e568de\" is forbidden: User \"system:node:ip-10-0-139-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-26.ec2.internal' and this object" podUID="6f10d14d-fb6a-49fd-9b04-0f616f192270" pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:38.820760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.820741    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a4809a-openshift-default-dockercfg-vxllr\""
Nov 02 14:19:38.938687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.938646    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-envoy\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:38.938898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.938794    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ba3232-d954-4fa1-a47b-3453590c7922-istiod-ca-cert\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:38.938898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.938847    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-socket\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:38.938977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.938897    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-certs\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:38.938977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.938933    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-podinfo\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:38.938977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.938962    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-credential-socket\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:38.939117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.939101    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-token\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:38.939195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.939171    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh7md\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-kube-api-access-hh7md\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:38.939327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:38.939233    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-data\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.040642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.040612    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-token\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.040861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.040772    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh7md\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-kube-api-access-hh7md\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.040983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.040875    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-data\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.040983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.040951    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-envoy\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041050    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ba3232-d954-4fa1-a47b-3453590c7922-istiod-ca-cert\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041086    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-socket\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041121    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-certs\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041150    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-podinfo\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041179    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-credential-socket\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041338    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-data\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041596    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-socket\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041663    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-certs\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041718    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-credential-socket\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.041877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.041858    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ba3232-d954-4fa1-a47b-3453590c7922-istiod-ca-cert\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.043693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.043660    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-envoy\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.043901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.043877    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-podinfo\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.067350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.067328    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh7md\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-kube-api-access-hh7md\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.067454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.067393    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-token\") pod \"e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") " pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.126810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.126736    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:39.398159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.398129    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1ff927-cad5-4b00-b3ae-39d9de8ce94c" path="/var/lib/kubelet/pods/2d1ff927-cad5-4b00-b3ae-39d9de8ce94c/volumes"
Nov 02 14:19:39.398627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.398606    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72df336e-41ed-45c2-b132-7c715685584e" path="/var/lib/kubelet/pods/72df336e-41ed-45c2-b132-7c715685584e/volumes"
Nov 02 14:19:39.399083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.399069    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" path="/var/lib/kubelet/pods/856cd9ab-9a6f-4ba3-b8ea-87b83d868832/volumes"
Nov 02 14:19:39.399828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.399810    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb" path="/var/lib/kubelet/pods/9fba0a38-2bbe-41ee-a8a6-8a7b8d7089eb/volumes"
Nov 02 14:19:39.406558 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:39.406531    5034 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7036-e568de\" is forbidden: User \"system:node:ip-10-0-139-26.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-26.ec2.internal' and this object" podUID="6f10d14d-fb6a-49fd-9b04-0f616f192270" pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:39.465368 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:19:39.465346    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ba3232_d954_4fa1_a47b_3453590c7922.slice/crio-00c67923f0f866f63a84eb091ffb81b2fd89aa805797862466a02b26049cbe5e WatchSource:0}: Error finding container 00c67923f0f866f63a84eb091ffb81b2fd89aa805797862466a02b26049cbe5e: Status 404 returned error can't find the container with id 00c67923f0f866f63a84eb091ffb81b2fd89aa805797862466a02b26049cbe5e
Nov 02 14:19:39.466764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.466726    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"]
Nov 02 14:19:39.467985 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.467956    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:19:39.468066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.468015    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:19:39.468066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.468050    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:19:39.512109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:39.512085    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" event={"ID":"f8ba3232-d954-4fa1-a47b-3453590c7922","Type":"ContainerStarted","Data":"00c67923f0f866f63a84eb091ffb81b2fd89aa805797862466a02b26049cbe5e"}
Nov 02 14:19:40.523636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:40.523605    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" event={"ID":"f8ba3232-d954-4fa1-a47b-3453590c7922","Type":"ContainerStarted","Data":"8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49"}
Nov 02 14:19:40.585533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:40.585475    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" podStartSLOduration=2.58546098 podStartE2EDuration="2.58546098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:40.584832963 +0000 UTC m=+2171.336137402" watchObservedRunningTime="2026-11-02 14:19:40.58546098 +0000 UTC m=+2171.336765417"
Nov 02 14:19:40.739653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:40.739629    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:40.871986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:40.871911    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f10d14d-fb6a-49fd-9b04-0f616f192270-kube-api-access-2h5gn\" (UniqueName: \"kubernetes.io/projected/6f10d14d-fb6a-49fd-9b04-0f616f192270-kube-api-access-2h5gn\") pod \"6f10d14d-fb6a-49fd-9b04-0f616f192270\" (UID: \"6f10d14d-fb6a-49fd-9b04-0f616f192270\") "
Nov 02 14:19:40.874048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:40.874025    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f10d14d-fb6a-49fd-9b04-0f616f192270-kube-api-access-2h5gn" pod "6f10d14d-fb6a-49fd-9b04-0f616f192270" (UID: "6f10d14d-fb6a-49fd-9b04-0f616f192270"). InnerVolumeSpecName "kube-api-access-2h5gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:19:40.988239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:40.988209    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h5gn\" (UniqueName: \"kubernetes.io/projected/6f10d14d-fb6a-49fd-9b04-0f616f192270-kube-api-access-2h5gn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:19:41.127445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.127376    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:41.133209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.133186    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:41.396236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.396166    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f10d14d-fb6a-49fd-9b04-0f616f192270" path="/var/lib/kubelet/pods/6f10d14d-fb6a-49fd-9b04-0f616f192270/volumes"
Nov 02 14:19:41.534316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.534280    5034 generic.go:356] "Generic (PLEG): container finished" podID="6f10d14d-fb6a-49fd-9b04-0f616f192270" containerID="c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835" exitCode=137
Nov 02 14:19:41.534954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.534351    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-e568de"
Nov 02 14:19:41.534954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.534378    5034 scope.go:122] "RemoveContainer" containerID="c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835"
Nov 02 14:19:41.534954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.534706    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:41.536424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.536403    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:19:41.546990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.546685    5034 scope.go:122] "RemoveContainer" containerID="c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835"
Nov 02 14:19:41.546990 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:41.546973    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835\": container with ID starting with c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835 not found: ID does not exist" containerID="c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835"
Nov 02 14:19:41.547125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:41.547017    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835"} err="failed to get container status \"c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835\": rpc error: code = NotFound desc = could not find container \"c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835\": container with ID starting with c66e736fb479c4b0a92835d24932af962b8a185d2cf4322e1be5b10231502835 not found: ID does not exist"
Nov 02 14:19:44.981417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:44.981385    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"]
Nov 02 14:19:44.982287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:44.982259    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="856cd9ab-9a6f-4ba3-b8ea-87b83d868832" containerName="istio-proxy"
Nov 02 14:19:44.987957 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:44.987939    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:45.013370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.013337    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"]
Nov 02 14:19:45.017520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.017498    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a4809a\""
Nov 02 14:19:45.059978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.059937    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpzfp\" (UniqueName: \"kubernetes.io/projected/06bc705b-a182-4a6f-b68c-5508e62e50b9-kube-api-access-hpzfp\") pod \"payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp\" (UID: \"06bc705b-a182-4a6f-b68c-5508e62e50b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:45.060213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.060191    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bc705b-a182-4a6f-b68c-5508e62e50b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp\" (UID: \"06bc705b-a182-4a6f-b68c-5508e62e50b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:45.117389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.117357    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"]
Nov 02 14:19:45.122472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.122453    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:45.172032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.171996    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a4809a-dockercfg-9g4f2\""
Nov 02 14:19:45.190744 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.190694    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpzfp\" (UniqueName: \"kubernetes.io/projected/06bc705b-a182-4a6f-b68c-5508e62e50b9-kube-api-access-hpzfp\") pod \"payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp\" (UID: \"06bc705b-a182-4a6f-b68c-5508e62e50b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:45.221066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.221026    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bc705b-a182-4a6f-b68c-5508e62e50b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp\" (UID: \"06bc705b-a182-4a6f-b68c-5508e62e50b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:45.232721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.222173    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bc705b-a182-4a6f-b68c-5508e62e50b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp\" (UID: \"06bc705b-a182-4a6f-b68c-5508e62e50b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:45.277895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.277844    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"]
Nov 02 14:19:45.332230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.332192    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpzfp\" (UniqueName: \"kubernetes.io/projected/06bc705b-a182-4a6f-b68c-5508e62e50b9-kube-api-access-hpzfp\") pod \"payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp\" (UID: \"06bc705b-a182-4a6f-b68c-5508e62e50b9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:45.366341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.366301    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspbh\" (UniqueName: \"kubernetes.io/projected/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-kube-api-access-jspbh\") pod \"payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn\" (UID: \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:45.366533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.366468    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn\" (UID: \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:45.467285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.467256    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn\" (UID: \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:45.467515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.467494    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspbh\" (UniqueName: \"kubernetes.io/projected/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-kube-api-access-jspbh\") pod \"payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn\" (UID: \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:45.468049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.468026    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn\" (UID: \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:45.505046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.504992    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspbh\" (UniqueName: \"kubernetes.io/projected/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-kube-api-access-jspbh\") pod \"payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn\" (UID: \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:45.564482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.564460    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:45.603294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.603262    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:45.880667 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:19:45.880638    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c0180a_6ee2_484d_99b2_0dd1ebfcd43d.slice/crio-9638a3ca4a80461ca188e83bc520e88bb3472a03688a84c47efeb6b87554cc72 WatchSource:0}: Error finding container 9638a3ca4a80461ca188e83bc520e88bb3472a03688a84c47efeb6b87554cc72: Status 404 returned error can't find the container with id 9638a3ca4a80461ca188e83bc520e88bb3472a03688a84c47efeb6b87554cc72
Nov 02 14:19:45.882815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.882761    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"]
Nov 02 14:19:45.929210 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:19:45.929180    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bc705b_a182_4a6f_b68c_5508e62e50b9.slice/crio-368457bae2987d17c41da1c98aa0f84c7425cf41e0acc2dccfe4e9152ce387a5 WatchSource:0}: Error finding container 368457bae2987d17c41da1c98aa0f84c7425cf41e0acc2dccfe4e9152ce387a5: Status 404 returned error can't find the container with id 368457bae2987d17c41da1c98aa0f84c7425cf41e0acc2dccfe4e9152ce387a5
Nov 02 14:19:45.930167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:45.930147    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"]
Nov 02 14:19:46.580288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:46.580249    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn" event={"ID":"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d","Type":"ContainerStarted","Data":"bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8"}
Nov 02 14:19:46.580801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:46.580312    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn" event={"ID":"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d","Type":"ContainerStarted","Data":"9638a3ca4a80461ca188e83bc520e88bb3472a03688a84c47efeb6b87554cc72"}
Nov 02 14:19:46.580801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:46.580346    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:46.582824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:46.582791    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp" event={"ID":"06bc705b-a182-4a6f-b68c-5508e62e50b9","Type":"ContainerStarted","Data":"e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89"}
Nov 02 14:19:46.582968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:46.582847    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp" event={"ID":"06bc705b-a182-4a6f-b68c-5508e62e50b9","Type":"ContainerStarted","Data":"368457bae2987d17c41da1c98aa0f84c7425cf41e0acc2dccfe4e9152ce387a5"}
Nov 02 14:19:46.583172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:46.583154    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:46.624929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:46.624884    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn" podStartSLOduration=1.6248722660000001 podStartE2EDuration="1.624872266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:46.6234403 +0000 UTC m=+2177.374744739" watchObservedRunningTime="2026-11-02 14:19:46.624872266 +0000 UTC m=+2177.376176703"
Nov 02 14:19:46.668022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:46.667976    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp" podStartSLOduration=2.667963419 podStartE2EDuration="2.667963419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:19:46.665830507 +0000 UTC m=+2177.417134948" watchObservedRunningTime="2026-11-02 14:19:46.667963419 +0000 UTC m=+2177.419267857"
Nov 02 14:19:48.415375 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:19:48.415334    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:19:57.597380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.597351    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:19:57.597826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.597413    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:19:57.694567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.694527    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"]
Nov 02 14:19:57.703093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.703067    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:57.724575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.723899    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:19:57.724575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.723971    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:19:57.724575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.724074    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:19:57.724575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.724184    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:19:57.729221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.729184    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"]
Nov 02 14:19:57.834996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.834954    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3f2936f-xnnbf\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:57.835190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.835036    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l99s\" (UniqueName: \"kubernetes.io/projected/9c34dd58-c602-4045-a31c-54e8bbec432f-kube-api-access-5l99s\") pod \"maas-api-revoke-sub-f3f2936f-xnnbf\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:57.835190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.835121    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3f2936f-xnnbf\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:57.936297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.936209    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3f2936f-xnnbf\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:57.936297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.936267    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l99s\" (UniqueName: \"kubernetes.io/projected/9c34dd58-c602-4045-a31c-54e8bbec432f-kube-api-access-5l99s\") pod \"maas-api-revoke-sub-f3f2936f-xnnbf\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:57.936545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.936317    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3f2936f-xnnbf\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:57.937037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.937007    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3f2936f-xnnbf\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:57.938874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.938851    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3f2936f-xnnbf\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:57.963397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:57.963376    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l99s\" (UniqueName: \"kubernetes.io/projected/9c34dd58-c602-4045-a31c-54e8bbec432f-kube-api-access-5l99s\") pod \"maas-api-revoke-sub-f3f2936f-xnnbf\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:58.024616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:58.024596    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:19:58.328848 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:19:58.328813    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c34dd58_c602_4045_a31c_54e8bbec432f.slice/crio-1a6e425d2614a2fa3665560210a1f4eb89c4835c5be4d284e483de23678bd7f2 WatchSource:0}: Error finding container 1a6e425d2614a2fa3665560210a1f4eb89c4835c5be4d284e483de23678bd7f2: Status 404 returned error can't find the container with id 1a6e425d2614a2fa3665560210a1f4eb89c4835c5be4d284e483de23678bd7f2
Nov 02 14:19:58.330413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:58.330385    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"]
Nov 02 14:19:58.714079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:58.714044    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf" event={"ID":"9c34dd58-c602-4045-a31c-54e8bbec432f","Type":"ContainerStarted","Data":"1a6e425d2614a2fa3665560210a1f4eb89c4835c5be4d284e483de23678bd7f2"}
Nov 02 14:19:59.723881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:59.723736    5034 generic.go:356] "Generic (PLEG): container finished" podID="9c34dd58-c602-4045-a31c-54e8bbec432f" containerID="a94e2b7937d0e0b06c941c2692dee37efa14b6ce160c9f1dc3f5413cf6039160" exitCode=0
Nov 02 14:19:59.723881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:19:59.723811    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf" event={"ID":"9c34dd58-c602-4045-a31c-54e8bbec432f","Type":"ContainerDied","Data":"a94e2b7937d0e0b06c941c2692dee37efa14b6ce160c9f1dc3f5413cf6039160"}
Nov 02 14:20:00.903134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:00.903105    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:20:00.980269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:00.980223    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c34dd58-c602-4045-a31c-54e8bbec432f-kube-api-access-5l99s\" (UniqueName: \"kubernetes.io/projected/9c34dd58-c602-4045-a31c-54e8bbec432f-kube-api-access-5l99s\") pod \"9c34dd58-c602-4045-a31c-54e8bbec432f\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") "
Nov 02 14:20:00.980479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:00.980376    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-xks-ca\") pod \"9c34dd58-c602-4045-a31c-54e8bbec432f\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") "
Nov 02 14:20:00.980479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:00.980468    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-openshift-ca\") pod \"9c34dd58-c602-4045-a31c-54e8bbec432f\" (UID: \"9c34dd58-c602-4045-a31c-54e8bbec432f\") "
Nov 02 14:20:00.981127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:00.981078    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-openshift-ca" pod "9c34dd58-c602-4045-a31c-54e8bbec432f" (UID: "9c34dd58-c602-4045-a31c-54e8bbec432f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:00.981298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:00.981279    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:00.987361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:00.987282    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c34dd58-c602-4045-a31c-54e8bbec432f-kube-api-access-5l99s" pod "9c34dd58-c602-4045-a31c-54e8bbec432f" (UID: "9c34dd58-c602-4045-a31c-54e8bbec432f"). InnerVolumeSpecName "kube-api-access-5l99s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:00.987361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:00.987311    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-xks-ca" pod "9c34dd58-c602-4045-a31c-54e8bbec432f" (UID: "9c34dd58-c602-4045-a31c-54e8bbec432f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:20:01.088205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:01.088167    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c34dd58-c602-4045-a31c-54e8bbec432f-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:01.088520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:01.088504    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l99s\" (UniqueName: \"kubernetes.io/projected/9c34dd58-c602-4045-a31c-54e8bbec432f-kube-api-access-5l99s\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:01.740334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:01.740305    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"
Nov 02 14:20:01.740559 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:01.740308    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf" event={"ID":"9c34dd58-c602-4045-a31c-54e8bbec432f","Type":"ContainerDied","Data":"1a6e425d2614a2fa3665560210a1f4eb89c4835c5be4d284e483de23678bd7f2"}
Nov 02 14:20:01.740559 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:01.740429    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a6e425d2614a2fa3665560210a1f4eb89c4835c5be4d284e483de23678bd7f2"
Nov 02 14:20:08.047417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.047384    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"]
Nov 02 14:20:08.048121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.048105    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9c34dd58-c602-4045-a31c-54e8bbec432f" containerName="revoke-keys"
Nov 02 14:20:08.048176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.048122    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c34dd58-c602-4045-a31c-54e8bbec432f" containerName="revoke-keys"
Nov 02 14:20:08.048273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.048264    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c34dd58-c602-4045-a31c-54e8bbec432f" containerName="revoke-keys"
Nov 02 14:20:08.052196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.052177    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.076720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.076680    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"]
Nov 02 14:20:08.084672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.084641    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:20:08.084841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.084710    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:20:08.084841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.084771    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:20:08.084994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.084882    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:20:08.138884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.138857    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"]
Nov 02 14:20:08.139253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.139224    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy" containerID="cri-o://87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d" gracePeriod=30
Nov 02 14:20:08.195796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.193945    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f151b97-hmfwd\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.195796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.194067    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f151b97-hmfwd\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.195796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.194102    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lpb\" (UniqueName: \"kubernetes.io/projected/4828403e-b64b-42f8-ad57-7c9db90c5c26-kube-api-access-q9lpb\") pod \"maas-api-revoke-sub-7f151b97-hmfwd\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.295854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.295815    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f151b97-hmfwd\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.296076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.295939    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f151b97-hmfwd\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.296076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.295977    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lpb\" (UniqueName: \"kubernetes.io/projected/4828403e-b64b-42f8-ad57-7c9db90c5c26-kube-api-access-q9lpb\") pod \"maas-api-revoke-sub-7f151b97-hmfwd\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.296600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.296566    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f151b97-hmfwd\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.299074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.299021    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f151b97-hmfwd\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.322969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.322930    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lpb\" (UniqueName: \"kubernetes.io/projected/4828403e-b64b-42f8-ad57-7c9db90c5c26-kube-api-access-q9lpb\") pod \"maas-api-revoke-sub-7f151b97-hmfwd\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.366651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.366618    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:08.714741 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:08.714693    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4828403e_b64b_42f8_ad57_7c9db90c5c26.slice/crio-2c661d5895c3c290b97cc910d0f5b3d8e3063242e94780322fa1a77f2ce915fa WatchSource:0}: Error finding container 2c661d5895c3c290b97cc910d0f5b3d8e3063242e94780322fa1a77f2ce915fa: Status 404 returned error can't find the container with id 2c661d5895c3c290b97cc910d0f5b3d8e3063242e94780322fa1a77f2ce915fa
Nov 02 14:20:08.716910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.716882    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"]
Nov 02 14:20:08.822529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:08.822476    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd" event={"ID":"4828403e-b64b-42f8-ad57-7c9db90c5c26","Type":"ContainerStarted","Data":"2c661d5895c3c290b97cc910d0f5b3d8e3063242e94780322fa1a77f2ce915fa"}
Nov 02 14:20:09.842947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:09.842846    5034 generic.go:356] "Generic (PLEG): container finished" podID="4828403e-b64b-42f8-ad57-7c9db90c5c26" containerID="c477f0e5db60f88644b2c7dec1fbc0199df1425d106b3f063d02b117643eaba4" exitCode=0
Nov 02 14:20:09.843438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:09.842944    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd" event={"ID":"4828403e-b64b-42f8-ad57-7c9db90c5c26","Type":"ContainerDied","Data":"c477f0e5db60f88644b2c7dec1fbc0199df1425d106b3f063d02b117643eaba4"}
Nov 02 14:20:11.023972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.023948    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:11.146941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.146864    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-xks-ca\") pod \"4828403e-b64b-42f8-ad57-7c9db90c5c26\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") "
Nov 02 14:20:11.147086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.146956    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4828403e-b64b-42f8-ad57-7c9db90c5c26-kube-api-access-q9lpb\" (UniqueName: \"kubernetes.io/projected/4828403e-b64b-42f8-ad57-7c9db90c5c26-kube-api-access-q9lpb\") pod \"4828403e-b64b-42f8-ad57-7c9db90c5c26\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") "
Nov 02 14:20:11.147086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.147047    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-openshift-ca\") pod \"4828403e-b64b-42f8-ad57-7c9db90c5c26\" (UID: \"4828403e-b64b-42f8-ad57-7c9db90c5c26\") "
Nov 02 14:20:11.147469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.147440    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-openshift-ca" pod "4828403e-b64b-42f8-ad57-7c9db90c5c26" (UID: "4828403e-b64b-42f8-ad57-7c9db90c5c26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:11.147603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.147584    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:11.149105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.149077    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4828403e-b64b-42f8-ad57-7c9db90c5c26-kube-api-access-q9lpb" pod "4828403e-b64b-42f8-ad57-7c9db90c5c26" (UID: "4828403e-b64b-42f8-ad57-7c9db90c5c26"). InnerVolumeSpecName "kube-api-access-q9lpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:11.149239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.149106    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-xks-ca" pod "4828403e-b64b-42f8-ad57-7c9db90c5c26" (UID: "4828403e-b64b-42f8-ad57-7c9db90c5c26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:20:11.248676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.248648    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4828403e-b64b-42f8-ad57-7c9db90c5c26-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:11.248676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.248676    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9lpb\" (UniqueName: \"kubernetes.io/projected/4828403e-b64b-42f8-ad57-7c9db90c5c26-kube-api-access-q9lpb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:11.861333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.861300    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"
Nov 02 14:20:11.861495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.861327    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd" event={"ID":"4828403e-b64b-42f8-ad57-7c9db90c5c26","Type":"ContainerDied","Data":"2c661d5895c3c290b97cc910d0f5b3d8e3063242e94780322fa1a77f2ce915fa"}
Nov 02 14:20:11.861495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:11.861391    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c661d5895c3c290b97cc910d0f5b3d8e3063242e94780322fa1a77f2ce915fa"
Nov 02 14:20:13.632913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.629182    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:20:13.739803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.733701    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b383071-be95-4a6c-b9b4-805bebc7c15e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b383071-be95-4a6c-b9b4-805bebc7c15e-istiod-ca-cert\") pod \"4b383071-be95-4a6c-b9b4-805bebc7c15e\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") "
Nov 02 14:20:13.739803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.733801    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-certs\") pod \"4b383071-be95-4a6c-b9b4-805bebc7c15e\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") "
Nov 02 14:20:13.739803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.733902    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-kube-api-access-9g22c\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-kube-api-access-9g22c\") pod \"4b383071-be95-4a6c-b9b4-805bebc7c15e\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") "
Nov 02 14:20:13.739803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.733982    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-data\") pod \"4b383071-be95-4a6c-b9b4-805bebc7c15e\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") "
Nov 02 14:20:13.739803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.734113    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-token\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-token\") pod \"4b383071-be95-4a6c-b9b4-805bebc7c15e\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") "
Nov 02 14:20:13.739803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.734164    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-envoy\") pod \"4b383071-be95-4a6c-b9b4-805bebc7c15e\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") "
Nov 02 14:20:13.739803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.734211    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-podinfo\") pod \"4b383071-be95-4a6c-b9b4-805bebc7c15e\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") "
Nov 02 14:20:13.739803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.734250    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-credential-socket\") pod \"4b383071-be95-4a6c-b9b4-805bebc7c15e\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") "
Nov 02 14:20:13.739803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.734302    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-socket\") pod \"4b383071-be95-4a6c-b9b4-805bebc7c15e\" (UID: \"4b383071-be95-4a6c-b9b4-805bebc7c15e\") "
Nov 02 14:20:13.740800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.740572    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-socket" pod "4b383071-be95-4a6c-b9b4-805bebc7c15e" (UID: "4b383071-be95-4a6c-b9b4-805bebc7c15e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:13.741302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.740969    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-certs" pod "4b383071-be95-4a6c-b9b4-805bebc7c15e" (UID: "4b383071-be95-4a6c-b9b4-805bebc7c15e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:13.743156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.743120    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b383071-be95-4a6c-b9b4-805bebc7c15e-istiod-ca-cert" pod "4b383071-be95-4a6c-b9b4-805bebc7c15e" (UID: "4b383071-be95-4a6c-b9b4-805bebc7c15e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:13.743461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.743436    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-data" pod "4b383071-be95-4a6c-b9b4-805bebc7c15e" (UID: "4b383071-be95-4a6c-b9b4-805bebc7c15e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:13.743788 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.743750    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-credential-socket" pod "4b383071-be95-4a6c-b9b4-805bebc7c15e" (UID: "4b383071-be95-4a6c-b9b4-805bebc7c15e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:13.748505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.747530    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-token" pod "4b383071-be95-4a6c-b9b4-805bebc7c15e" (UID: "4b383071-be95-4a6c-b9b4-805bebc7c15e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:13.748505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.748088    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-podinfo" pod "4b383071-be95-4a6c-b9b4-805bebc7c15e" (UID: "4b383071-be95-4a6c-b9b4-805bebc7c15e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:20:13.748505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.748453    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-envoy" pod "4b383071-be95-4a6c-b9b4-805bebc7c15e" (UID: "4b383071-be95-4a6c-b9b4-805bebc7c15e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:13.750162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.750126    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-kube-api-access-9g22c" pod "4b383071-be95-4a6c-b9b4-805bebc7c15e" (UID: "4b383071-be95-4a6c-b9b4-805bebc7c15e"). InnerVolumeSpecName "kube-api-access-9g22c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:13.836027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.835988    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:13.836027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.836015    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:13.836027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.836027    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:13.836374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.836041    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b383071-be95-4a6c-b9b4-805bebc7c15e-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:13.836374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.836053    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:13.836374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.836061    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:13.836374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.836070    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b383071-be95-4a6c-b9b4-805bebc7c15e-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:13.836374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.836078    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b383071-be95-4a6c-b9b4-805bebc7c15e-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:13.836374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.836086    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g22c\" (UniqueName: \"kubernetes.io/projected/4b383071-be95-4a6c-b9b4-805bebc7c15e-kube-api-access-9g22c\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:13.885423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.885377    5034 generic.go:356] "Generic (PLEG): container finished" podID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerID="87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d" exitCode=0
Nov 02 14:20:13.885628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.885449    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"
Nov 02 14:20:13.885628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.885466    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" event={"ID":"4b383071-be95-4a6c-b9b4-805bebc7c15e","Type":"ContainerDied","Data":"87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d"}
Nov 02 14:20:13.885628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.885559    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq" event={"ID":"4b383071-be95-4a6c-b9b4-805bebc7c15e","Type":"ContainerDied","Data":"71be742b6bd68a08bf059e1c8a97dd5ddd2d9514488fb3790c37b25923ba2885"}
Nov 02 14:20:13.885628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.885589    5034 scope.go:122] "RemoveContainer" containerID="87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d"
Nov 02 14:20:13.899033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.899010    5034 scope.go:122] "RemoveContainer" containerID="87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d"
Nov 02 14:20:13.899312 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:20:13.899294    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d\": container with ID starting with 87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d not found: ID does not exist" containerID="87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d"
Nov 02 14:20:13.899389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.899319    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d"} err="failed to get container status \"87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d\": rpc error: code = NotFound desc = could not find container \"87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d\": container with ID starting with 87ef2dd13581153dbe6d74575bf99778a3d59b4f5949c40d17aca4de73de349d not found: ID does not exist"
Nov 02 14:20:13.945159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.945130    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"]
Nov 02 14:20:13.956710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:13.956679    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-v8sqq"]
Nov 02 14:20:14.033148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:14.033060    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"]
Nov 02 14:20:14.033320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:14.033301    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy" containerID="cri-o://65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4" gracePeriod=30
Nov 02 14:20:15.399146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.399114    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" path="/var/lib/kubelet/pods/4b383071-be95-4a6c-b9b4-805bebc7c15e/volumes"
Nov 02 14:20:15.417665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.417629    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"]
Nov 02 14:20:15.418428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.418407    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4828403e-b64b-42f8-ad57-7c9db90c5c26" containerName="revoke-keys"
Nov 02 14:20:15.418428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.418428    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4828403e-b64b-42f8-ad57-7c9db90c5c26" containerName="revoke-keys"
Nov 02 14:20:15.418628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.418444    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy"
Nov 02 14:20:15.418628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.418451    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy"
Nov 02 14:20:15.418734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.418680    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4828403e-b64b-42f8-ad57-7c9db90c5c26" containerName="revoke-keys"
Nov 02 14:20:15.418734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.418695    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b383071-be95-4a6c-b9b4-805bebc7c15e" containerName="istio-proxy"
Nov 02 14:20:15.423822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.423801    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.446273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.446248    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"]
Nov 02 14:20:15.452408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.452372    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:20:15.452521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.452452    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:20:15.453178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.453157    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:20:15.453393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.453371    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:20:15.458997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.458974    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d237115e-k8twg\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.459100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.459026    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d237115e-k8twg\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.459314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.459291    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhfrh\" (UniqueName: \"kubernetes.io/projected/445dc6a8-5650-472e-8028-d9824aa09880-kube-api-access-jhfrh\") pod \"maas-api-revoke-sub-d237115e-k8twg\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.560423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.560390    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhfrh\" (UniqueName: \"kubernetes.io/projected/445dc6a8-5650-472e-8028-d9824aa09880-kube-api-access-jhfrh\") pod \"maas-api-revoke-sub-d237115e-k8twg\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.560568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.560494    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d237115e-k8twg\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.560568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.560534    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d237115e-k8twg\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.561161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.561140    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d237115e-k8twg\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.562964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.562944    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d237115e-k8twg\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.584763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.584736    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhfrh\" (UniqueName: \"kubernetes.io/projected/445dc6a8-5650-472e-8028-d9824aa09880-kube-api-access-jhfrh\") pod \"maas-api-revoke-sub-d237115e-k8twg\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:15.738377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:15.738350    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:16.086950 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:16.086919    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445dc6a8_5650_472e_8028_d9824aa09880.slice/crio-66fb775bf0a70ac46214a1c75dc7d52921a1610878f504e693d82c55f742bab7 WatchSource:0}: Error finding container 66fb775bf0a70ac46214a1c75dc7d52921a1610878f504e693d82c55f742bab7: Status 404 returned error can't find the container with id 66fb775bf0a70ac46214a1c75dc7d52921a1610878f504e693d82c55f742bab7
Nov 02 14:20:16.089728 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:16.089697    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"]
Nov 02 14:20:16.918152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:16.918121    5034 generic.go:356] "Generic (PLEG): container finished" podID="445dc6a8-5650-472e-8028-d9824aa09880" containerID="f9f0a528d4f1134316acbdf183f30cbafd4003cdee02b68be9e6557e32cd3ee6" exitCode=0
Nov 02 14:20:16.918481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:16.918208    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg" event={"ID":"445dc6a8-5650-472e-8028-d9824aa09880","Type":"ContainerDied","Data":"f9f0a528d4f1134316acbdf183f30cbafd4003cdee02b68be9e6557e32cd3ee6"}
Nov 02 14:20:16.918481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:16.918265    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg" event={"ID":"445dc6a8-5650-472e-8028-d9824aa09880","Type":"ContainerStarted","Data":"66fb775bf0a70ac46214a1c75dc7d52921a1610878f504e693d82c55f742bab7"}
Nov 02 14:20:18.117932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.117904    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:18.206839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.206713    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445dc6a8-5650-472e-8028-d9824aa09880-kube-api-access-jhfrh\" (UniqueName: \"kubernetes.io/projected/445dc6a8-5650-472e-8028-d9824aa09880-kube-api-access-jhfrh\") pod \"445dc6a8-5650-472e-8028-d9824aa09880\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") "
Nov 02 14:20:18.207374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.207336    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-openshift-ca\") pod \"445dc6a8-5650-472e-8028-d9824aa09880\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") "
Nov 02 14:20:18.207665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.207644    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-xks-ca\") pod \"445dc6a8-5650-472e-8028-d9824aa09880\" (UID: \"445dc6a8-5650-472e-8028-d9824aa09880\") "
Nov 02 14:20:18.208147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.207865    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-openshift-ca" pod "445dc6a8-5650-472e-8028-d9824aa09880" (UID: "445dc6a8-5650-472e-8028-d9824aa09880"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:18.208919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.208894    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:18.218841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.210838    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445dc6a8-5650-472e-8028-d9824aa09880-kube-api-access-jhfrh" pod "445dc6a8-5650-472e-8028-d9824aa09880" (UID: "445dc6a8-5650-472e-8028-d9824aa09880"). InnerVolumeSpecName "kube-api-access-jhfrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:18.220075 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.220033    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-xks-ca" pod "445dc6a8-5650-472e-8028-d9824aa09880" (UID: "445dc6a8-5650-472e-8028-d9824aa09880"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:20:18.310020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.309984    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445dc6a8-5650-472e-8028-d9824aa09880-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:18.310020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.310016    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhfrh\" (UniqueName: \"kubernetes.io/projected/445dc6a8-5650-472e-8028-d9824aa09880-kube-api-access-jhfrh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:18.415980 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:20:18.415934    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:20:18.934867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.934829    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"
Nov 02 14:20:18.934867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.934848    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg" event={"ID":"445dc6a8-5650-472e-8028-d9824aa09880","Type":"ContainerDied","Data":"66fb775bf0a70ac46214a1c75dc7d52921a1610878f504e693d82c55f742bab7"}
Nov 02 14:20:18.935110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:18.934908    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66fb775bf0a70ac46214a1c75dc7d52921a1610878f504e693d82c55f742bab7"
Nov 02 14:20:19.330209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.330182    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:20:19.424715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.424687    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-certs\") pod \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") "
Nov 02 14:20:19.424918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.424721    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-socket\") pod \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") "
Nov 02 14:20:19.424918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.424839    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istiod-ca-cert\") pod \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") "
Nov 02 14:20:19.425025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.424929    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-token\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-token\") pod \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") "
Nov 02 14:20:19.425025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.424962    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-credential-socket\") pod \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") "
Nov 02 14:20:19.425025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.425002    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-certs" pod "44d79516-f3b6-46ed-b3bc-e43bfc44465f" (UID: "44d79516-f3b6-46ed-b3bc-e43bfc44465f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:19.425196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.425086    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-podinfo\") pod \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") "
Nov 02 14:20:19.425196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.425110    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-socket" pod "44d79516-f3b6-46ed-b3bc-e43bfc44465f" (UID: "44d79516-f3b6-46ed-b3bc-e43bfc44465f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:19.425196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.425133    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-envoy\") pod \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") "
Nov 02 14:20:19.425373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.425209    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-kube-api-access-tmm54\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-kube-api-access-tmm54\") pod \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") "
Nov 02 14:20:19.425373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.425238    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-data\") pod \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\" (UID: \"44d79516-f3b6-46ed-b3bc-e43bfc44465f\") "
Nov 02 14:20:19.425373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.425285    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-credential-socket" pod "44d79516-f3b6-46ed-b3bc-e43bfc44465f" (UID: "44d79516-f3b6-46ed-b3bc-e43bfc44465f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:19.425373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.425300    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istiod-ca-cert" pod "44d79516-f3b6-46ed-b3bc-e43bfc44465f" (UID: "44d79516-f3b6-46ed-b3bc-e43bfc44465f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:19.425645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.425622    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-data" pod "44d79516-f3b6-46ed-b3bc-e43bfc44465f" (UID: "44d79516-f3b6-46ed-b3bc-e43bfc44465f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:19.426394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.426163    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:19.426394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.426188    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:19.426394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.426203    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:19.426394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.426216    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:19.426394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.426230    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:19.427699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.427670    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-token" pod "44d79516-f3b6-46ed-b3bc-e43bfc44465f" (UID: "44d79516-f3b6-46ed-b3bc-e43bfc44465f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:19.427870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.427848    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-podinfo" pod "44d79516-f3b6-46ed-b3bc-e43bfc44465f" (UID: "44d79516-f3b6-46ed-b3bc-e43bfc44465f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:20:19.427952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.427934    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-envoy" pod "44d79516-f3b6-46ed-b3bc-e43bfc44465f" (UID: "44d79516-f3b6-46ed-b3bc-e43bfc44465f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:19.428071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.428053    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-kube-api-access-tmm54" pod "44d79516-f3b6-46ed-b3bc-e43bfc44465f" (UID: "44d79516-f3b6-46ed-b3bc-e43bfc44465f"). InnerVolumeSpecName "kube-api-access-tmm54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:19.527645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.527615    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:19.527645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.527643    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:19.527645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.527652    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d79516-f3b6-46ed-b3bc-e43bfc44465f-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:19.527942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.527661    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmm54\" (UniqueName: \"kubernetes.io/projected/44d79516-f3b6-46ed-b3bc-e43bfc44465f-kube-api-access-tmm54\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:19.945818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.945719    5034 generic.go:356] "Generic (PLEG): container finished" podID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerID="65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4" exitCode=0
Nov 02 14:20:19.945818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.945766    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"
Nov 02 14:20:19.945818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.945808    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" event={"ID":"44d79516-f3b6-46ed-b3bc-e43bfc44465f","Type":"ContainerDied","Data":"65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4"}
Nov 02 14:20:19.946122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.945869    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc" event={"ID":"44d79516-f3b6-46ed-b3bc-e43bfc44465f","Type":"ContainerDied","Data":"233cc2431bb70bf76d837251e723c7e8a0b3df0ce8a13566ad5f625f962b4563"}
Nov 02 14:20:19.946122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.945896    5034 scope.go:122] "RemoveContainer" containerID="65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4"
Nov 02 14:20:19.958200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.958182    5034 scope.go:122] "RemoveContainer" containerID="65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4"
Nov 02 14:20:19.958487 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:20:19.958467    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4\": container with ID starting with 65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4 not found: ID does not exist" containerID="65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4"
Nov 02 14:20:19.958544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:19.958498    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4"} err="failed to get container status \"65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4\": rpc error: code = NotFound desc = could not find container \"65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4\": container with ID starting with 65be499efa2b6b94229788646ed8c84ee915cbddcd2ee98e8b6e635ca7b785f4 not found: ID does not exist"
Nov 02 14:20:20.024138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:20.024111    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"]
Nov 02 14:20:20.028188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:20.028167    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-rblsc"]
Nov 02 14:20:20.223947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:20.221169    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"]
Nov 02 14:20:20.442465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:20.442428    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"]
Nov 02 14:20:20.442969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:20.442704    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy" containerID="cri-o://c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716" gracePeriod=30
Nov 02 14:20:21.397240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:21.397206    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" path="/var/lib/kubelet/pods/44d79516-f3b6-46ed-b3bc-e43bfc44465f/volumes"
Nov 02 14:20:25.734764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.734738    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 14:20:25.741833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.741808    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"]
Nov 02 14:20:25.742503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.742487    5034 cpu_manager.go:414] "containerMap: removing container" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy"
Nov 02 14:20:25.742578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.742504    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy"
Nov 02 14:20:25.742578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.742529    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy"
Nov 02 14:20:25.742578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.742534    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy"
Nov 02 14:20:25.742578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.742561    5034 cpu_manager.go:414] "containerMap: removing container" podUID="445dc6a8-5650-472e-8028-d9824aa09880" containerName="revoke-keys"
Nov 02 14:20:25.742578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.742565    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445dc6a8-5650-472e-8028-d9824aa09880" containerName="revoke-keys"
Nov 02 14:20:25.742899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.742701    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy"
Nov 02 14:20:25.742899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.742711    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d79516-f3b6-46ed-b3bc-e43bfc44465f" containerName="istio-proxy"
Nov 02 14:20:25.742899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.742724    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="445dc6a8-5650-472e-8028-d9824aa09880" containerName="revoke-keys"
Nov 02 14:20:25.746170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.746153    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:25.764547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.764518    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:20:25.764677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.764663    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:20:25.767216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.767180    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"]
Nov 02 14:20:25.784189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.784170    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:20:25.784542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.784488    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:20:25.825626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.825600    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-certs\") pod \"cb1af079-7c3c-444f-a426-b716bde35270\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") "
Nov 02 14:20:25.825905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.825888    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-data\") pod \"cb1af079-7c3c-444f-a426-b716bde35270\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") "
Nov 02 14:20:25.826036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.826022    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1af079-7c3c-444f-a426-b716bde35270-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1af079-7c3c-444f-a426-b716bde35270-istiod-ca-cert\") pod \"cb1af079-7c3c-444f-a426-b716bde35270\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") "
Nov 02 14:20:25.826172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.826158    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-socket\") pod \"cb1af079-7c3c-444f-a426-b716bde35270\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") "
Nov 02 14:20:25.826367 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.826354    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-kube-api-access-c4txx\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-kube-api-access-c4txx\") pod \"cb1af079-7c3c-444f-a426-b716bde35270\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") "
Nov 02 14:20:25.826530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.826517    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-envoy\") pod \"cb1af079-7c3c-444f-a426-b716bde35270\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") "
Nov 02 14:20:25.826638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.826626    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-istio-token\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-istio-token\") pod \"cb1af079-7c3c-444f-a426-b716bde35270\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") "
Nov 02 14:20:25.826761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.826739    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-credential-socket\") pod \"cb1af079-7c3c-444f-a426-b716bde35270\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") "
Nov 02 14:20:25.827316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.827298    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb1af079-7c3c-444f-a426-b716bde35270-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1af079-7c3c-444f-a426-b716bde35270-istio-podinfo\") pod \"cb1af079-7c3c-444f-a426-b716bde35270\" (UID: \"cb1af079-7c3c-444f-a426-b716bde35270\") "
Nov 02 14:20:25.828113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.828089    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ccc56c-b977-4711-8940-c7101b802009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d4da17-j2qdd\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:25.828405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.828388    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ccc56c-b977-4711-8940-c7101b802009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d4da17-j2qdd\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:25.828715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.828697    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt47x\" (UniqueName: \"kubernetes.io/projected/03ccc56c-b977-4711-8940-c7101b802009-kube-api-access-qt47x\") pod \"maas-api-revoke-sub-05d4da17-j2qdd\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:25.829338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.829314    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-certs" pod "cb1af079-7c3c-444f-a426-b716bde35270" (UID: "cb1af079-7c3c-444f-a426-b716bde35270"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:25.829521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.829504    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-data" pod "cb1af079-7c3c-444f-a426-b716bde35270" (UID: "cb1af079-7c3c-444f-a426-b716bde35270"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:25.829833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.829769    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1af079-7c3c-444f-a426-b716bde35270-istiod-ca-cert" pod "cb1af079-7c3c-444f-a426-b716bde35270" (UID: "cb1af079-7c3c-444f-a426-b716bde35270"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:25.830048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.830029    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-socket" pod "cb1af079-7c3c-444f-a426-b716bde35270" (UID: "cb1af079-7c3c-444f-a426-b716bde35270"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:25.830515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.830490    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-credential-socket" pod "cb1af079-7c3c-444f-a426-b716bde35270" (UID: "cb1af079-7c3c-444f-a426-b716bde35270"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:25.839714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.839690    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-istio-token" pod "cb1af079-7c3c-444f-a426-b716bde35270" (UID: "cb1af079-7c3c-444f-a426-b716bde35270"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:25.840759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.840734    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-kube-api-access-c4txx" pod "cb1af079-7c3c-444f-a426-b716bde35270" (UID: "cb1af079-7c3c-444f-a426-b716bde35270"). InnerVolumeSpecName "kube-api-access-c4txx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:25.840912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.840874    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-envoy" pod "cb1af079-7c3c-444f-a426-b716bde35270" (UID: "cb1af079-7c3c-444f-a426-b716bde35270"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:25.843352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.843327    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb1af079-7c3c-444f-a426-b716bde35270-istio-podinfo" pod "cb1af079-7c3c-444f-a426-b716bde35270" (UID: "cb1af079-7c3c-444f-a426-b716bde35270"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:20:25.930729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.930693    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ccc56c-b977-4711-8940-c7101b802009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d4da17-j2qdd\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:25.930978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.930826    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ccc56c-b977-4711-8940-c7101b802009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d4da17-j2qdd\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:25.930978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.930936    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt47x\" (UniqueName: \"kubernetes.io/projected/03ccc56c-b977-4711-8940-c7101b802009-kube-api-access-qt47x\") pod \"maas-api-revoke-sub-05d4da17-j2qdd\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:25.931116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931042    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb1af079-7c3c-444f-a426-b716bde35270-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:25.931116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931055    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:25.931116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931063    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:25.931116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931074    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb1af079-7c3c-444f-a426-b716bde35270-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:25.931116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931087    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:25.931116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931097    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4txx\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-kube-api-access-c4txx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:25.931116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931106    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:25.931116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931113    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb1af079-7c3c-444f-a426-b716bde35270-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:25.931116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931121    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb1af079-7c3c-444f-a426-b716bde35270-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:25.931645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.931614    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ccc56c-b977-4711-8940-c7101b802009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d4da17-j2qdd\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:25.933333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.933308    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ccc56c-b977-4711-8940-c7101b802009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d4da17-j2qdd\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:25.965687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:25.965634    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt47x\" (UniqueName: \"kubernetes.io/projected/03ccc56c-b977-4711-8940-c7101b802009-kube-api-access-qt47x\") pod \"maas-api-revoke-sub-05d4da17-j2qdd\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:26.004166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.004139    5034 generic.go:356] "Generic (PLEG): container finished" podID="cb1af079-7c3c-444f-a426-b716bde35270" containerID="c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716" exitCode=0
Nov 02 14:20:26.004276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.004187    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"
Nov 02 14:20:26.004276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.004229    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" event={"ID":"cb1af079-7c3c-444f-a426-b716bde35270","Type":"ContainerDied","Data":"c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716"}
Nov 02 14:20:26.004381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.004294    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp" event={"ID":"cb1af079-7c3c-444f-a426-b716bde35270","Type":"ContainerDied","Data":"d67e91b95c1190fc36392c089514bc39896e460ea21a14a5a86f57403f4621bc"}
Nov 02 14:20:26.004381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.004328    5034 scope.go:122] "RemoveContainer" containerID="c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716"
Nov 02 14:20:26.015794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.015758    5034 scope.go:122] "RemoveContainer" containerID="daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e"
Nov 02 14:20:26.028211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.028188    5034 scope.go:122] "RemoveContainer" containerID="c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716"
Nov 02 14:20:26.028507 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:20:26.028482    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716\": container with ID starting with c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716 not found: ID does not exist" containerID="c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716"
Nov 02 14:20:26.028602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.028519    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716"} err="failed to get container status \"c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716\": rpc error: code = NotFound desc = could not find container \"c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716\": container with ID starting with c99fad87fb837873dc00709f90625afb8a913997c89eeea9a7cf607849d7c716 not found: ID does not exist"
Nov 02 14:20:26.028602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.028545    5034 scope.go:122] "RemoveContainer" containerID="daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e"
Nov 02 14:20:26.028841 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:20:26.028824    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e\": container with ID starting with daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e not found: ID does not exist" containerID="daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e"
Nov 02 14:20:26.028902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.028849    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e"} err="failed to get container status \"daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e\": rpc error: code = NotFound desc = could not find container \"daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e\": container with ID starting with daed211ae6ea3137d7e3f34628ce278e41f150d0dbcc8489b2b26026720c946e not found: ID does not exist"
Nov 02 14:20:26.068806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.065025    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:26.106817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.102074    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"]
Nov 02 14:20:26.108240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.108208    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9f9xp"]
Nov 02 14:20:26.412064 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:26.412019    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ccc56c_b977_4711_8940_c7101b802009.slice/crio-4f25461e8512fc01985e7a69ae923a7ac94a9287232e547aa8d4751640e82287 WatchSource:0}: Error finding container 4f25461e8512fc01985e7a69ae923a7ac94a9287232e547aa8d4751640e82287: Status 404 returned error can't find the container with id 4f25461e8512fc01985e7a69ae923a7ac94a9287232e547aa8d4751640e82287
Nov 02 14:20:26.412835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:26.412765    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"]
Nov 02 14:20:27.016596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:27.016562    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd" event={"ID":"03ccc56c-b977-4711-8940-c7101b802009","Type":"ContainerStarted","Data":"ee5b239c6a2d57d14c2fd6a5da21416276f0be0f90628885c3002fb28527432d"}
Nov 02 14:20:27.017052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:27.016619    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd" event={"ID":"03ccc56c-b977-4711-8940-c7101b802009","Type":"ContainerStarted","Data":"4f25461e8512fc01985e7a69ae923a7ac94a9287232e547aa8d4751640e82287"}
Nov 02 14:20:27.063648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:27.063560    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd" podStartSLOduration=2.063547989 podStartE2EDuration="2.063547989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:27.059861715 +0000 UTC m=+2217.811166156" watchObservedRunningTime="2026-11-02 14:20:27.063547989 +0000 UTC m=+2217.814852474"
Nov 02 14:20:27.398058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:27.398022    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1af079-7c3c-444f-a426-b716bde35270" path="/var/lib/kubelet/pods/cb1af079-7c3c-444f-a426-b716bde35270/volumes"
Nov 02 14:20:28.026921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:28.026885    5034 generic.go:356] "Generic (PLEG): container finished" podID="03ccc56c-b977-4711-8940-c7101b802009" containerID="ee5b239c6a2d57d14c2fd6a5da21416276f0be0f90628885c3002fb28527432d" exitCode=0
Nov 02 14:20:28.027394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:28.026985    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd" event={"ID":"03ccc56c-b977-4711-8940-c7101b802009","Type":"ContainerDied","Data":"ee5b239c6a2d57d14c2fd6a5da21416276f0be0f90628885c3002fb28527432d"}
Nov 02 14:20:29.246628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.246599    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:29.378577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.378503    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ccc56c-b977-4711-8940-c7101b802009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ccc56c-b977-4711-8940-c7101b802009-maas-api-xks-ca\") pod \"03ccc56c-b977-4711-8940-c7101b802009\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") "
Nov 02 14:20:29.378577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.378536    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ccc56c-b977-4711-8940-c7101b802009-kube-api-access-qt47x\" (UniqueName: \"kubernetes.io/projected/03ccc56c-b977-4711-8940-c7101b802009-kube-api-access-qt47x\") pod \"03ccc56c-b977-4711-8940-c7101b802009\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") "
Nov 02 14:20:29.378799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.378649    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ccc56c-b977-4711-8940-c7101b802009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ccc56c-b977-4711-8940-c7101b802009-maas-api-openshift-ca\") pod \"03ccc56c-b977-4711-8940-c7101b802009\" (UID: \"03ccc56c-b977-4711-8940-c7101b802009\") "
Nov 02 14:20:29.379024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.378980    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ccc56c-b977-4711-8940-c7101b802009-maas-api-openshift-ca" pod "03ccc56c-b977-4711-8940-c7101b802009" (UID: "03ccc56c-b977-4711-8940-c7101b802009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:29.379270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.379252    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03ccc56c-b977-4711-8940-c7101b802009-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:29.380581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.380552    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ccc56c-b977-4711-8940-c7101b802009-maas-api-xks-ca" pod "03ccc56c-b977-4711-8940-c7101b802009" (UID: "03ccc56c-b977-4711-8940-c7101b802009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:20:29.380802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.380768    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ccc56c-b977-4711-8940-c7101b802009-kube-api-access-qt47x" pod "03ccc56c-b977-4711-8940-c7101b802009" (UID: "03ccc56c-b977-4711-8940-c7101b802009"). InnerVolumeSpecName "kube-api-access-qt47x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:29.480883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.480852    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03ccc56c-b977-4711-8940-c7101b802009-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:29.480883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:29.480884    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt47x\" (UniqueName: \"kubernetes.io/projected/03ccc56c-b977-4711-8940-c7101b802009-kube-api-access-qt47x\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:30.058820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.058762    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"
Nov 02 14:20:30.059022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.058829    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd" event={"ID":"03ccc56c-b977-4711-8940-c7101b802009","Type":"ContainerDied","Data":"4f25461e8512fc01985e7a69ae923a7ac94a9287232e547aa8d4751640e82287"}
Nov 02 14:20:30.059022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.058881    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f25461e8512fc01985e7a69ae923a7ac94a9287232e547aa8d4751640e82287"
Nov 02 14:20:30.162471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.162432    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"]
Nov 02 14:20:30.163156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.163140    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy"
Nov 02 14:20:30.163156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.163156    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy"
Nov 02 14:20:30.163262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.163166    5034 cpu_manager.go:414] "containerMap: removing container" podUID="03ccc56c-b977-4711-8940-c7101b802009" containerName="revoke-keys"
Nov 02 14:20:30.163262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.163171    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ccc56c-b977-4711-8940-c7101b802009" containerName="revoke-keys"
Nov 02 14:20:30.163329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.163288    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ccc56c-b977-4711-8940-c7101b802009" containerName="revoke-keys"
Nov 02 14:20:30.163329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.163307    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1af079-7c3c-444f-a426-b716bde35270" containerName="istio-proxy"
Nov 02 14:20:30.167128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.167100    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:30.185424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.183957    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-26b508\""
Nov 02 14:20:30.185424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.183993    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-26b508-dockercfg-8t2cj\""
Nov 02 14:20:30.191011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.190979    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"]
Nov 02 14:20:30.304896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.304665    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkqlz\" (UniqueName: \"kubernetes.io/projected/0604c248-3e5a-4de8-8772-932e884a4326-kube-api-access-qkqlz\") pod \"payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh\" (UID: \"0604c248-3e5a-4de8-8772-932e884a4326\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:30.346932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.346757    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0604c248-3e5a-4de8-8772-932e884a4326-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh\" (UID: \"0604c248-3e5a-4de8-8772-932e884a4326\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:30.449616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.449577    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkqlz\" (UniqueName: \"kubernetes.io/projected/0604c248-3e5a-4de8-8772-932e884a4326-kube-api-access-qkqlz\") pod \"payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh\" (UID: \"0604c248-3e5a-4de8-8772-932e884a4326\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:30.449844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.449821    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0604c248-3e5a-4de8-8772-932e884a4326-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh\" (UID: \"0604c248-3e5a-4de8-8772-932e884a4326\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:30.450581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.450562    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0604c248-3e5a-4de8-8772-932e884a4326-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh\" (UID: \"0604c248-3e5a-4de8-8772-932e884a4326\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:30.474185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.474161    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkqlz\" (UniqueName: \"kubernetes.io/projected/0604c248-3e5a-4de8-8772-932e884a4326-kube-api-access-qkqlz\") pod \"payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh\" (UID: \"0604c248-3e5a-4de8-8772-932e884a4326\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:30.482045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.482027    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:30.790655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:30.790358    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"]
Nov 02 14:20:30.791901 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:30.791878    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0604c248_3e5a_4de8_8772_932e884a4326.slice/crio-579044597e7aaa86143fd809b3d9dce62d37de39f809cd8ef76e5beb40b06940 WatchSource:0}: Error finding container 579044597e7aaa86143fd809b3d9dce62d37de39f809cd8ef76e5beb40b06940: Status 404 returned error can't find the container with id 579044597e7aaa86143fd809b3d9dce62d37de39f809cd8ef76e5beb40b06940
Nov 02 14:20:31.069177 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:31.069078    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh" event={"ID":"0604c248-3e5a-4de8-8772-932e884a4326","Type":"ContainerStarted","Data":"613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f"}
Nov 02 14:20:31.069177 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:31.069145    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh" event={"ID":"0604c248-3e5a-4de8-8772-932e884a4326","Type":"ContainerStarted","Data":"579044597e7aaa86143fd809b3d9dce62d37de39f809cd8ef76e5beb40b06940"}
Nov 02 14:20:31.069432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:31.069272    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:31.116091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:31.116031    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh" podStartSLOduration=1.116012735 podStartE2EDuration="1.116012735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:31.111499741 +0000 UTC m=+2221.862804182" watchObservedRunningTime="2026-11-02 14:20:31.116012735 +0000 UTC m=+2221.867317174"
Nov 02 14:20:34.530307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:34.530274    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"]
Nov 02 14:20:35.248323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.248283    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf" podUID="bee64526-add5-4d0b-a53f-f420ba83d8ba" containerName="main" containerID="cri-o://0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5" gracePeriod=45
Nov 02 14:20:35.545471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.545441    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:20:35.646402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.646358    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-model-cache\") pod \"bee64526-add5-4d0b-a53f-f420ba83d8ba\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") "
Nov 02 14:20:35.646631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.646429    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-home\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-home\") pod \"bee64526-add5-4d0b-a53f-f420ba83d8ba\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") "
Nov 02 14:20:35.646631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.646517    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-tmp-dir\") pod \"bee64526-add5-4d0b-a53f-f420ba83d8ba\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") "
Nov 02 14:20:35.646631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.646592    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee64526-add5-4d0b-a53f-f420ba83d8ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/bee64526-add5-4d0b-a53f-f420ba83d8ba-tls-certs\") pod \"bee64526-add5-4d0b-a53f-f420ba83d8ba\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") "
Nov 02 14:20:35.646865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.646685    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-dshm\") pod \"bee64526-add5-4d0b-a53f-f420ba83d8ba\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") "
Nov 02 14:20:35.646865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.646755    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee64526-add5-4d0b-a53f-f420ba83d8ba-kube-api-access-btlqk\" (UniqueName: \"kubernetes.io/projected/bee64526-add5-4d0b-a53f-f420ba83d8ba-kube-api-access-btlqk\") pod \"bee64526-add5-4d0b-a53f-f420ba83d8ba\" (UID: \"bee64526-add5-4d0b-a53f-f420ba83d8ba\") "
Nov 02 14:20:35.677799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.659871    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-model-cache" pod "bee64526-add5-4d0b-a53f-f420ba83d8ba" (UID: "bee64526-add5-4d0b-a53f-f420ba83d8ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:35.677799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.664866    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-home" pod "bee64526-add5-4d0b-a53f-f420ba83d8ba" (UID: "bee64526-add5-4d0b-a53f-f420ba83d8ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:35.711802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.691416    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-tmp-dir" pod "bee64526-add5-4d0b-a53f-f420ba83d8ba" (UID: "bee64526-add5-4d0b-a53f-f420ba83d8ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:35.711802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.691572    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee64526-add5-4d0b-a53f-f420ba83d8ba-kube-api-access-btlqk" pod "bee64526-add5-4d0b-a53f-f420ba83d8ba" (UID: "bee64526-add5-4d0b-a53f-f420ba83d8ba"). InnerVolumeSpecName "kube-api-access-btlqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:35.728632 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.728599    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-dshm" pod "bee64526-add5-4d0b-a53f-f420ba83d8ba" (UID: "bee64526-add5-4d0b-a53f-f420ba83d8ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:35.730305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.730257    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee64526-add5-4d0b-a53f-f420ba83d8ba-tls-certs" pod "bee64526-add5-4d0b-a53f-f420ba83d8ba" (UID: "bee64526-add5-4d0b-a53f-f420ba83d8ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:20:35.751801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.749276    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:35.751801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.749315    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee64526-add5-4d0b-a53f-f420ba83d8ba-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:35.751801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.749329    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:35.751801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.749345    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btlqk\" (UniqueName: \"kubernetes.io/projected/bee64526-add5-4d0b-a53f-f420ba83d8ba-kube-api-access-btlqk\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:35.751801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.749358    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:35.751801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:35.749370    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee64526-add5-4d0b-a53f-f420ba83d8ba-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:36.127467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.127433    5034 generic.go:356] "Generic (PLEG): container finished" podID="bee64526-add5-4d0b-a53f-f420ba83d8ba" containerID="0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5" exitCode=0
Nov 02 14:20:36.127648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.127477    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"
Nov 02 14:20:36.127648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.127514    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf" event={"ID":"bee64526-add5-4d0b-a53f-f420ba83d8ba","Type":"ContainerDied","Data":"0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5"}
Nov 02 14:20:36.127648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.127557    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf" event={"ID":"bee64526-add5-4d0b-a53f-f420ba83d8ba","Type":"ContainerDied","Data":"1a333beb60f74f487ce40a18a36cf30d9452c998bbcb1dfd9ae0f7e508f64893"}
Nov 02 14:20:36.127648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.127574    5034 scope.go:122] "RemoveContainer" containerID="0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5"
Nov 02 14:20:36.138673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.138654    5034 scope.go:122] "RemoveContainer" containerID="0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5"
Nov 02 14:20:36.138983 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:20:36.138959    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5\": container with ID starting with 0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5 not found: ID does not exist" containerID="0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5"
Nov 02 14:20:36.139103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.138987    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5"} err="failed to get container status \"0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5\": rpc error: code = NotFound desc = could not find container \"0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5\": container with ID starting with 0902b977c563b5e45ab17e706171c3ade4e8102048a04d79a25e6730984a33b5 not found: ID does not exist"
Nov 02 14:20:36.188563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.188539    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"]
Nov 02 14:20:36.201360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.201331    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-960a68/auth-test-model-960a68-kserve-ddbc55bc7-4xjrf"]
Nov 02 14:20:36.371679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.371643    5034 scope.go:122] "RemoveContainer" containerID="0879910e666fb336013ca6e998458eb6241b6cf51573661a09a41edb5b5ba851"
Nov 02 14:20:36.383444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.383422    5034 scope.go:122] "RemoveContainer" containerID="3d9c5d05c9dd7a871edc8528934da0fe9bfc6b07247b346d2078f26edbd2f0de"
Nov 02 14:20:36.393452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.393418    5034 scope.go:122] "RemoveContainer" containerID="2c1c4475cc3eedee7daf6febba775ebf1b68216214cbff5f08452981bb09758d"
Nov 02 14:20:36.403078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.403047    5034 scope.go:122] "RemoveContainer" containerID="ea73fa8b4e9364a649445bdc3a680ae289bdd3b6badcf8a1169fe3333719b931"
Nov 02 14:20:36.412471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.412442    5034 scope.go:122] "RemoveContainer" containerID="33549632a99f551df2578c2f84cc6aef7a495b50f0090a6af93e336910c4a25d"
Nov 02 14:20:36.478655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.478629    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"]
Nov 02 14:20:36.478866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.478840    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy" containerID="cri-o://a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004" gracePeriod=30
Nov 02 14:20:36.544855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.544823    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"]
Nov 02 14:20:36.545913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.545893    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bee64526-add5-4d0b-a53f-f420ba83d8ba" containerName="main"
Nov 02 14:20:36.546217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.545916    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee64526-add5-4d0b-a53f-f420ba83d8ba" containerName="main"
Nov 02 14:20:36.546217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.546126    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee64526-add5-4d0b-a53f-f420ba83d8ba" containerName="main"
Nov 02 14:20:36.615098 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.614436    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"]
Nov 02 14:20:36.615098 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.614547    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.632433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.632403    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-5f98a6-openshift-default-dockercfg-4785q\""
Nov 02 14:20:36.779712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.779667    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.779712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.779718    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-certs\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.780008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.779795    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-token\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.780081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.779993    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-credential-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.780081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.780036    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25wv\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-kube-api-access-m25wv\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.780199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.780079    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istiod-ca-cert\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.780199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.780180    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-envoy\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.780311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.780248    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-podinfo\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.780311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.780289    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-data\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.881582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.881552    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.881834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.881589    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-certs\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.881834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.881690    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-token\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.881910    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-credential-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.881959    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25wv\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-kube-api-access-m25wv\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.882023    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.882006    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istiod-ca-cert\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.882090    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-certs\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.882200    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-envoy\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.882309    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-credential-socket\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.882320    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-podinfo\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.882383    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-data\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.882749    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istiod-ca-cert\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.882825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.882799    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-data\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.884619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.884593    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-envoy\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.885011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.884990    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-podinfo\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.906696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.906669    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-token\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.906981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.906962    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25wv\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-kube-api-access-m25wv\") pod \"e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") " pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:36.940853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:36.940828    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:37.272341 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:37.272299    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e1dc5d3_a4ad_42f7_9e88_fe4b3a4fe9d9.slice/crio-7e8799b8be951a34bacf4a8f7c8189faa136f1c712b0e19037877432e1759171 WatchSource:0}: Error finding container 7e8799b8be951a34bacf4a8f7c8189faa136f1c712b0e19037877432e1759171: Status 404 returned error can't find the container with id 7e8799b8be951a34bacf4a8f7c8189faa136f1c712b0e19037877432e1759171
Nov 02 14:20:37.273407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:37.273378    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"]
Nov 02 14:20:37.276070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:37.276026    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:20:37.276228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:37.276117    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:20:37.276228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:37.276161    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:20:37.401488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:37.401458    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee64526-add5-4d0b-a53f-f420ba83d8ba" path="/var/lib/kubelet/pods/bee64526-add5-4d0b-a53f-f420ba83d8ba/volumes"
Nov 02 14:20:38.143730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:38.143698    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" event={"ID":"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9","Type":"ContainerStarted","Data":"f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60"}
Nov 02 14:20:38.144258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:38.143752    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" event={"ID":"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9","Type":"ContainerStarted","Data":"7e8799b8be951a34bacf4a8f7c8189faa136f1c712b0e19037877432e1759171"}
Nov 02 14:20:38.219579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:38.219525    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" podStartSLOduration=2.2195107370000002 podStartE2EDuration="2.219510737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:38.20428824 +0000 UTC m=+2228.955592678" watchObservedRunningTime="2026-11-02 14:20:38.219510737 +0000 UTC m=+2228.970815174"
Nov 02 14:20:38.941941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:38.941912    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:38.948299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:38.948272    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:39.151819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:39.151767    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:39.152762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:39.152742    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:20:40.350133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.347255    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"]
Nov 02 14:20:40.354063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.354039    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.387016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.386985    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-662e3e-openshift-default-dockercfg-9j7cs\""
Nov 02 14:20:40.407683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.407044    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"]
Nov 02 14:20:40.493938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.460899    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af80e624-1231-4cfd-8024-64373c76b18d-istiod-ca-cert\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.493938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.460955    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-socket\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.493938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.460982    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-credential-socket\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.493938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.461017    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-envoy\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.493938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.461078    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-istio-token\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.493938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.461102    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxw5\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-kube-api-access-npxw5\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.493938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.461147    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-data\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.493938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.461401    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-certs\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.493938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.461462    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af80e624-1231-4cfd-8024-64373c76b18d-istio-podinfo\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.608258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.602709    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-certs\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.615330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.608868    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-certs\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.615612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.615592    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af80e624-1231-4cfd-8024-64373c76b18d-istio-podinfo\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.616303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.616280    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af80e624-1231-4cfd-8024-64373c76b18d-istiod-ca-cert\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.626427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.617487    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af80e624-1231-4cfd-8024-64373c76b18d-istiod-ca-cert\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.626876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.626831    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-socket\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.633870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.626371    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-socket\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.638830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.636879    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af80e624-1231-4cfd-8024-64373c76b18d-istio-podinfo\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.638830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.637636    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-credential-socket\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.639074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.639050    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-credential-socket\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.639195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.639184    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-envoy\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.639326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.639297    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-istio-token\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.639419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.639408    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxw5\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-kube-api-access-npxw5\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.639511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.639501    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-data\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.639922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.639886    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-data\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.642006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.641985    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-envoy\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.668400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.668370    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxw5\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-kube-api-access-npxw5\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.669942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.669916    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-istio-token\") pod \"e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") " pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:40.674930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:40.674909    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:41.061219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:41.056145    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"]
Nov 02 14:20:41.064796 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:41.064155    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf80e624_1231_4cfd_8024_64373c76b18d.slice/crio-244a563a26492f269a52f20e82cea6953ecd67abd5734d4f6181a279fa63d558 WatchSource:0}: Error finding container 244a563a26492f269a52f20e82cea6953ecd67abd5734d4f6181a279fa63d558: Status 404 returned error can't find the container with id 244a563a26492f269a52f20e82cea6953ecd67abd5734d4f6181a279fa63d558
Nov 02 14:20:41.070004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:41.068401    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:20:41.070004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:41.068500    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:20:41.070004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:41.068596    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:20:41.203432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:41.201911    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" event={"ID":"af80e624-1231-4cfd-8024-64373c76b18d","Type":"ContainerStarted","Data":"244a563a26492f269a52f20e82cea6953ecd67abd5734d4f6181a279fa63d558"}
Nov 02 14:20:42.068366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.068327    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:20:42.096800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.096073    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:20:42.223081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.217914    5034 generic.go:356] "Generic (PLEG): container finished" podID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerID="a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004" exitCode=0
Nov 02 14:20:42.223081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.217975    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"
Nov 02 14:20:42.223081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.218104    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" event={"ID":"1b3c337c-f1fb-44cb-856d-1f86fc8d908b","Type":"ContainerDied","Data":"a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004"}
Nov 02 14:20:42.223081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.218157    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh" event={"ID":"1b3c337c-f1fb-44cb-856d-1f86fc8d908b","Type":"ContainerDied","Data":"30b442c0cc5e28d4131938ae0a08d8d36044795ed482a24f39eaec324f6009cc"}
Nov 02 14:20:42.223081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.218189    5034 scope.go:122] "RemoveContainer" containerID="a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004"
Nov 02 14:20:42.223081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.222178    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" event={"ID":"af80e624-1231-4cfd-8024-64373c76b18d","Type":"ContainerStarted","Data":"db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca"}
Nov 02 14:20:42.231795 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.231749    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-credential-socket\") pod \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") "
Nov 02 14:20:42.232017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.231994    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-credential-socket" pod "1b3c337c-f1fb-44cb-856d-1f86fc8d908b" (UID: "1b3c337c-f1fb-44cb-856d-1f86fc8d908b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:42.232112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.232002    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-envoy\") pod \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") "
Nov 02 14:20:42.232112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.232073    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-certs\") pod \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") "
Nov 02 14:20:42.232241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.232145    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-podinfo\") pod \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") "
Nov 02 14:20:42.232241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.232231    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istiod-ca-cert\") pod \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") "
Nov 02 14:20:42.232359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.232288    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-token\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-token\") pod \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") "
Nov 02 14:20:42.232417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.232371    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-data\") pod \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") "
Nov 02 14:20:42.232497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.232479    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-socket\") pod \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") "
Nov 02 14:20:42.232614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.232598    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-kube-api-access-42zkx\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-kube-api-access-42zkx\") pod \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\" (UID: \"1b3c337c-f1fb-44cb-856d-1f86fc8d908b\") "
Nov 02 14:20:42.234896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.232822    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-certs" pod "1b3c337c-f1fb-44cb-856d-1f86fc8d908b" (UID: "1b3c337c-f1fb-44cb-856d-1f86fc8d908b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:42.234896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.233335    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:42.234896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.233360    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:42.234896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.234371    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istiod-ca-cert" pod "1b3c337c-f1fb-44cb-856d-1f86fc8d908b" (UID: "1b3c337c-f1fb-44cb-856d-1f86fc8d908b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:42.234896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.234804    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-socket" pod "1b3c337c-f1fb-44cb-856d-1f86fc8d908b" (UID: "1b3c337c-f1fb-44cb-856d-1f86fc8d908b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:42.234896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.234865    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-data" pod "1b3c337c-f1fb-44cb-856d-1f86fc8d908b" (UID: "1b3c337c-f1fb-44cb-856d-1f86fc8d908b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:42.243254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.240124    5034 scope.go:122] "RemoveContainer" containerID="a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004"
Nov 02 14:20:42.243254 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:20:42.240642    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004\": container with ID starting with a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004 not found: ID does not exist" containerID="a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004"
Nov 02 14:20:42.243254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.240707    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004"} err="failed to get container status \"a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004\": rpc error: code = NotFound desc = could not find container \"a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004\": container with ID starting with a4274eb5fc9b079375bcce86761d05cda8ed807083ef485fa303141b5d2d6004 not found: ID does not exist"
Nov 02 14:20:42.243254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.243075    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-kube-api-access-42zkx" pod "1b3c337c-f1fb-44cb-856d-1f86fc8d908b" (UID: "1b3c337c-f1fb-44cb-856d-1f86fc8d908b"). InnerVolumeSpecName "kube-api-access-42zkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:42.244722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.244693    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-envoy" pod "1b3c337c-f1fb-44cb-856d-1f86fc8d908b" (UID: "1b3c337c-f1fb-44cb-856d-1f86fc8d908b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:42.244876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.244827    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-podinfo" pod "1b3c337c-f1fb-44cb-856d-1f86fc8d908b" (UID: "1b3c337c-f1fb-44cb-856d-1f86fc8d908b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:20:42.246886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.246863    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-token" pod "1b3c337c-f1fb-44cb-856d-1f86fc8d908b" (UID: "1b3c337c-f1fb-44cb-856d-1f86fc8d908b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:42.334896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.334863    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:42.334896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.334896    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:42.335167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.334911    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:42.335167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.334926    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:42.335167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.334938    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:42.335167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.334950    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:42.335167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.334963    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42zkx\" (UniqueName: \"kubernetes.io/projected/1b3c337c-f1fb-44cb-856d-1f86fc8d908b-kube-api-access-42zkx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:42.593808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.592967    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" podStartSLOduration=2.592942732 podStartE2EDuration="2.592942732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:42.29301107 +0000 UTC m=+2233.044315508" watchObservedRunningTime="2026-11-02 14:20:42.592942732 +0000 UTC m=+2233.344247171"
Nov 02 14:20:42.607858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.595508    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"]
Nov 02 14:20:42.614456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.614426    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-cjdkh"]
Nov 02 14:20:42.677804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.676098    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:42.700805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:42.699088    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:43.232690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:43.232658    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:43.233920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:43.233898    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:20:43.405086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:43.405054    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" path="/var/lib/kubelet/pods/1b3c337c-f1fb-44cb-856d-1f86fc8d908b/volumes"
Nov 02 14:20:45.226926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.226887    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"]
Nov 02 14:20:45.227915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.227892    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy"
Nov 02 14:20:45.227915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.227915    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy"
Nov 02 14:20:45.228140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.228125    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3c337c-f1fb-44cb-856d-1f86fc8d908b" containerName="istio-proxy"
Nov 02 14:20:45.232956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.232926    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.251167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.251141    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"]
Nov 02 14:20:45.259888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.259853    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:20:45.264518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.264493    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:20:45.264736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.264717    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:20:45.264842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.264826    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:20:45.412888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.412847    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dece2c2-t26m9\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.413072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.412924    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dece2c2-t26m9\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.413072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.413022    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdjw\" (UniqueName: \"kubernetes.io/projected/0f108e3a-366e-43a8-a0be-72ad201fde3d-kube-api-access-jxdjw\") pod \"maas-api-revoke-sub-1dece2c2-t26m9\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.514649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.514609    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdjw\" (UniqueName: \"kubernetes.io/projected/0f108e3a-366e-43a8-a0be-72ad201fde3d-kube-api-access-jxdjw\") pod \"maas-api-revoke-sub-1dece2c2-t26m9\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.514949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.514925    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dece2c2-t26m9\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.515070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.515040    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dece2c2-t26m9\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.515700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.515676    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dece2c2-t26m9\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.517496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.517467    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dece2c2-t26m9\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.543118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.543093    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdjw\" (UniqueName: \"kubernetes.io/projected/0f108e3a-366e-43a8-a0be-72ad201fde3d-kube-api-access-jxdjw\") pod \"maas-api-revoke-sub-1dece2c2-t26m9\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.547444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.547428    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:45.922283 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:45.922253    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f108e3a_366e_43a8_a0be_72ad201fde3d.slice/crio-b04ea03723ec3a93f5052f7084d36f51e1bd912f9ee30cd36f141c3329396fac WatchSource:0}: Error finding container b04ea03723ec3a93f5052f7084d36f51e1bd912f9ee30cd36f141c3329396fac: Status 404 returned error can't find the container with id b04ea03723ec3a93f5052f7084d36f51e1bd912f9ee30cd36f141c3329396fac
Nov 02 14:20:45.924832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:45.924808    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"]
Nov 02 14:20:46.256943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.256909    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9" event={"ID":"0f108e3a-366e-43a8-a0be-72ad201fde3d","Type":"ContainerStarted","Data":"b04ea03723ec3a93f5052f7084d36f51e1bd912f9ee30cd36f141c3329396fac"}
Nov 02 14:20:46.563571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.563484    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"]
Nov 02 14:20:46.568613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.568589    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:46.610806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.608960    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-662e3e-dockercfg-s5tk2\""
Nov 02 14:20:46.615861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.615831    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"]
Nov 02 14:20:46.631926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.631425    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfjmj\" (UniqueName: \"kubernetes.io/projected/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-kube-api-access-vfjmj\") pod \"payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95\" (UID: \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:46.631926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.631556    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95\" (UID: \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:46.652127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.652093    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-662e3e\""
Nov 02 14:20:46.760803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.750050    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfjmj\" (UniqueName: \"kubernetes.io/projected/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-kube-api-access-vfjmj\") pod \"payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95\" (UID: \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:46.760803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.750237    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95\" (UID: \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:46.760803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.751263    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95\" (UID: \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:46.851604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.795744    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfjmj\" (UniqueName: \"kubernetes.io/projected/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-kube-api-access-vfjmj\") pod \"payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95\" (UID: \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:46.933264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:46.933232    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:47.294862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:47.293298    5034 generic.go:356] "Generic (PLEG): container finished" podID="0f108e3a-366e-43a8-a0be-72ad201fde3d" containerID="de158d18fbb6251b37826e6a57e3c1ec9ca8cd4ccd73716886187cead5b0a894" exitCode=0
Nov 02 14:20:47.294862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:47.293345    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9" event={"ID":"0f108e3a-366e-43a8-a0be-72ad201fde3d","Type":"ContainerDied","Data":"de158d18fbb6251b37826e6a57e3c1ec9ca8cd4ccd73716886187cead5b0a894"}
Nov 02 14:20:47.322592 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:47.322555    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95588e4f_c3d8_4cd7_acea_b4efb1b457c7.slice/crio-fc8b49a9c7d1fa8447d815f6487dd933e8c7c889aa15779c412c8a3f6c41b224 WatchSource:0}: Error finding container fc8b49a9c7d1fa8447d815f6487dd933e8c7c889aa15779c412c8a3f6c41b224: Status 404 returned error can't find the container with id fc8b49a9c7d1fa8447d815f6487dd933e8c7c889aa15779c412c8a3f6c41b224
Nov 02 14:20:47.324635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:47.324606    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"]
Nov 02 14:20:48.304307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.304269    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95" event={"ID":"95588e4f-c3d8-4cd7-acea-b4efb1b457c7","Type":"ContainerStarted","Data":"8119ba0643256a9041a041550bcf0cbbf5aae3040a880ac507c7ce0d1cd62d8c"}
Nov 02 14:20:48.304888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.304431    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95" event={"ID":"95588e4f-c3d8-4cd7-acea-b4efb1b457c7","Type":"ContainerStarted","Data":"fc8b49a9c7d1fa8447d815f6487dd933e8c7c889aa15779c412c8a3f6c41b224"}
Nov 02 14:20:48.304888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.304611    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:48.356100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.356050    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95" podStartSLOduration=2.356035399 podStartE2EDuration="2.356035399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:48.352724417 +0000 UTC m=+2239.104028855" watchObservedRunningTime="2026-11-02 14:20:48.356035399 +0000 UTC m=+2239.107339837"
Nov 02 14:20:48.436800 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:20:48.436340    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:20:48.543693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.543658    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:48.634268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.634178    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-xks-ca\") pod \"0f108e3a-366e-43a8-a0be-72ad201fde3d\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") "
Nov 02 14:20:48.634268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.634223    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f108e3a-366e-43a8-a0be-72ad201fde3d-kube-api-access-jxdjw\" (UniqueName: \"kubernetes.io/projected/0f108e3a-366e-43a8-a0be-72ad201fde3d-kube-api-access-jxdjw\") pod \"0f108e3a-366e-43a8-a0be-72ad201fde3d\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") "
Nov 02 14:20:48.634574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.634552    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-openshift-ca\") pod \"0f108e3a-366e-43a8-a0be-72ad201fde3d\" (UID: \"0f108e3a-366e-43a8-a0be-72ad201fde3d\") "
Nov 02 14:20:48.635032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.634974    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-openshift-ca" pod "0f108e3a-366e-43a8-a0be-72ad201fde3d" (UID: "0f108e3a-366e-43a8-a0be-72ad201fde3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:48.635427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.635395    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:48.650502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.650469    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-xks-ca" pod "0f108e3a-366e-43a8-a0be-72ad201fde3d" (UID: "0f108e3a-366e-43a8-a0be-72ad201fde3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:20:48.650874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.650846    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f108e3a-366e-43a8-a0be-72ad201fde3d-kube-api-access-jxdjw" pod "0f108e3a-366e-43a8-a0be-72ad201fde3d" (UID: "0f108e3a-366e-43a8-a0be-72ad201fde3d"). InnerVolumeSpecName "kube-api-access-jxdjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:48.737310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.737271    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f108e3a-366e-43a8-a0be-72ad201fde3d-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:48.737310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:48.737306    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdjw\" (UniqueName: \"kubernetes.io/projected/0f108e3a-366e-43a8-a0be-72ad201fde3d-kube-api-access-jxdjw\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:49.318755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.318723    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"
Nov 02 14:20:49.319349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.318760    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9" event={"ID":"0f108e3a-366e-43a8-a0be-72ad201fde3d","Type":"ContainerDied","Data":"b04ea03723ec3a93f5052f7084d36f51e1bd912f9ee30cd36f141c3329396fac"}
Nov 02 14:20:49.319349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.318825    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04ea03723ec3a93f5052f7084d36f51e1bd912f9ee30cd36f141c3329396fac"
Nov 02 14:20:49.552001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.551947    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8" podUID="96422e8c-4df5-48aa-8b50-b871af7fddd7" containerName="main" containerID="cri-o://9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5" gracePeriod=45
Nov 02 14:20:49.894327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.894299    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:20:49.954980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.954948    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-model-cache\") pod \"96422e8c-4df5-48aa-8b50-b871af7fddd7\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") "
Nov 02 14:20:49.955170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.955032    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-home\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-home\") pod \"96422e8c-4df5-48aa-8b50-b871af7fddd7\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") "
Nov 02 14:20:49.955170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.955054    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-tmp-dir\") pod \"96422e8c-4df5-48aa-8b50-b871af7fddd7\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") "
Nov 02 14:20:49.955170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.955120    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96422e8c-4df5-48aa-8b50-b871af7fddd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/96422e8c-4df5-48aa-8b50-b871af7fddd7-tls-certs\") pod \"96422e8c-4df5-48aa-8b50-b871af7fddd7\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") "
Nov 02 14:20:49.955394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.955187    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96422e8c-4df5-48aa-8b50-b871af7fddd7-kube-api-access-csdq2\" (UniqueName: \"kubernetes.io/projected/96422e8c-4df5-48aa-8b50-b871af7fddd7-kube-api-access-csdq2\") pod \"96422e8c-4df5-48aa-8b50-b871af7fddd7\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") "
Nov 02 14:20:49.955394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.955216    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-dshm\") pod \"96422e8c-4df5-48aa-8b50-b871af7fddd7\" (UID: \"96422e8c-4df5-48aa-8b50-b871af7fddd7\") "
Nov 02 14:20:49.956271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.956233    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-tmp-dir" pod "96422e8c-4df5-48aa-8b50-b871af7fddd7" (UID: "96422e8c-4df5-48aa-8b50-b871af7fddd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:49.956722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.956432    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-model-cache" pod "96422e8c-4df5-48aa-8b50-b871af7fddd7" (UID: "96422e8c-4df5-48aa-8b50-b871af7fddd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:49.956722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.956467    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-home" pod "96422e8c-4df5-48aa-8b50-b871af7fddd7" (UID: "96422e8c-4df5-48aa-8b50-b871af7fddd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:49.958981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.958952    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-dshm" pod "96422e8c-4df5-48aa-8b50-b871af7fddd7" (UID: "96422e8c-4df5-48aa-8b50-b871af7fddd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:20:49.959536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.959507    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96422e8c-4df5-48aa-8b50-b871af7fddd7-tls-certs" pod "96422e8c-4df5-48aa-8b50-b871af7fddd7" (UID: "96422e8c-4df5-48aa-8b50-b871af7fddd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:20:49.959860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:49.959823    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96422e8c-4df5-48aa-8b50-b871af7fddd7-kube-api-access-csdq2" pod "96422e8c-4df5-48aa-8b50-b871af7fddd7" (UID: "96422e8c-4df5-48aa-8b50-b871af7fddd7"). InnerVolumeSpecName "kube-api-access-csdq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:50.057537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.057499    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:50.057537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.057538    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:50.057720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.057553    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96422e8c-4df5-48aa-8b50-b871af7fddd7-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:50.057720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.057568    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csdq2\" (UniqueName: \"kubernetes.io/projected/96422e8c-4df5-48aa-8b50-b871af7fddd7-kube-api-access-csdq2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:50.057720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.057580    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:50.057720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.057594    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96422e8c-4df5-48aa-8b50-b871af7fddd7-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:50.327413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.327374    5034 generic.go:356] "Generic (PLEG): container finished" podID="96422e8c-4df5-48aa-8b50-b871af7fddd7" containerID="9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5" exitCode=0
Nov 02 14:20:50.327987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.327422    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"
Nov 02 14:20:50.327987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.327446    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8" event={"ID":"96422e8c-4df5-48aa-8b50-b871af7fddd7","Type":"ContainerDied","Data":"9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5"}
Nov 02 14:20:50.327987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.327490    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8" event={"ID":"96422e8c-4df5-48aa-8b50-b871af7fddd7","Type":"ContainerDied","Data":"46d1414958e52c044a1171c2cd02f0dc684de1d55c2f66ac4a8d37addb1152fd"}
Nov 02 14:20:50.327987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.327515    5034 scope.go:122] "RemoveContainer" containerID="9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5"
Nov 02 14:20:50.339124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.339078    5034 scope.go:122] "RemoveContainer" containerID="9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5"
Nov 02 14:20:50.339337 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:20:50.339317    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5\": container with ID starting with 9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5 not found: ID does not exist" containerID="9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5"
Nov 02 14:20:50.339421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.339345    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5"} err="failed to get container status \"9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5\": rpc error: code = NotFound desc = could not find container \"9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5\": container with ID starting with 9d876b6ed54a13bee7a06c6ecccec92ad8e38606d93063a151f0ca5fd88165f5 not found: ID does not exist"
Nov 02 14:20:50.390177 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.390140    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"]
Nov 02 14:20:50.393969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:50.393935    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-fcefcc/auth-test-model-fcefcc-kserve-64b9f56f-qkxf8"]
Nov 02 14:20:51.337973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:51.337899    5034 generic.go:356] "Generic (PLEG): container finished" podID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerID="8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49" exitCode=0
Nov 02 14:20:51.338494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:51.337966    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" event={"ID":"f8ba3232-d954-4fa1-a47b-3453590c7922","Type":"ContainerDied","Data":"8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49"}
Nov 02 14:20:51.338494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:51.338435    5034 scope.go:122] "RemoveContainer" containerID="8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49"
Nov 02 14:20:51.340398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:51.340373    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:20:51.340503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:51.340440    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:20:51.340503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:51.340473    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:20:51.407578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:51.407547    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96422e8c-4df5-48aa-8b50-b871af7fddd7" path="/var/lib/kubelet/pods/96422e8c-4df5-48aa-8b50-b871af7fddd7/volumes"
Nov 02 14:20:52.350051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:52.350014    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" event={"ID":"f8ba3232-d954-4fa1-a47b-3453590c7922","Type":"ContainerStarted","Data":"d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564"}
Nov 02 14:20:53.127164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:53.127131    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:20:53.128203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:53.128173    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 14:20:53.128361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:53.128220    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 14:20:54.127895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.127861    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 14:20:54.128354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.127915    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 14:20:54.597948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.597906    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"]
Nov 02 14:20:54.598691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.598668    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0f108e3a-366e-43a8-a0be-72ad201fde3d" containerName="revoke-keys"
Nov 02 14:20:54.598874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.598848    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f108e3a-366e-43a8-a0be-72ad201fde3d" containerName="revoke-keys"
Nov 02 14:20:54.599003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.598982    5034 cpu_manager.go:414] "containerMap: removing container" podUID="96422e8c-4df5-48aa-8b50-b871af7fddd7" containerName="main"
Nov 02 14:20:54.599081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.599006    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96422e8c-4df5-48aa-8b50-b871af7fddd7" containerName="main"
Nov 02 14:20:54.599206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.599193    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="96422e8c-4df5-48aa-8b50-b871af7fddd7" containerName="main"
Nov 02 14:20:54.599267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.599218    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f108e3a-366e-43a8-a0be-72ad201fde3d" containerName="revoke-keys"
Nov 02 14:20:54.603025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.603003    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:20:54.612058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.612035    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"]
Nov 02 14:20:54.621063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.621021    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-5f98a6\""
Nov 02 14:20:54.621358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.621319    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-5f98a6-dockercfg-7qrwm\""
Nov 02 14:20:54.849057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.844529    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rlpq\" (UniqueName: \"kubernetes.io/projected/4ac001bf-2ca9-4400-a244-f4404bf2351c-kube-api-access-4rlpq\") pod \"payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz\" (UID: \"4ac001bf-2ca9-4400-a244-f4404bf2351c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:20:54.850431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.850403    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac001bf-2ca9-4400-a244-f4404bf2351c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz\" (UID: \"4ac001bf-2ca9-4400-a244-f4404bf2351c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:20:54.858496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.858466    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"]
Nov 02 14:20:54.903187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.903155    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"]
Nov 02 14:20:54.903434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.903267    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:20:54.952169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.952142    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6c78f1-4137-4146-958b-726d999c64b5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs\" (UID: \"5b6c78f1-4137-4146-958b-726d999c64b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:20:54.952326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.952195    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlpq\" (UniqueName: \"kubernetes.io/projected/4ac001bf-2ca9-4400-a244-f4404bf2351c-kube-api-access-4rlpq\") pod \"payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz\" (UID: \"4ac001bf-2ca9-4400-a244-f4404bf2351c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:20:54.952326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.952305    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rv9g\" (UniqueName: \"kubernetes.io/projected/5b6c78f1-4137-4146-958b-726d999c64b5-kube-api-access-7rv9g\") pod \"payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs\" (UID: \"5b6c78f1-4137-4146-958b-726d999c64b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:20:54.952542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.952518    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac001bf-2ca9-4400-a244-f4404bf2351c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz\" (UID: \"4ac001bf-2ca9-4400-a244-f4404bf2351c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:20:54.953166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.953148    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac001bf-2ca9-4400-a244-f4404bf2351c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz\" (UID: \"4ac001bf-2ca9-4400-a244-f4404bf2351c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:20:54.996434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:54.996403    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rlpq\" (UniqueName: \"kubernetes.io/projected/4ac001bf-2ca9-4400-a244-f4404bf2351c-kube-api-access-4rlpq\") pod \"payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz\" (UID: \"4ac001bf-2ca9-4400-a244-f4404bf2351c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:20:55.054389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.054359    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6c78f1-4137-4146-958b-726d999c64b5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs\" (UID: \"5b6c78f1-4137-4146-958b-726d999c64b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:20:55.054546 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.054455    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rv9g\" (UniqueName: \"kubernetes.io/projected/5b6c78f1-4137-4146-958b-726d999c64b5-kube-api-access-7rv9g\") pod \"payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs\" (UID: \"5b6c78f1-4137-4146-958b-726d999c64b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:20:55.055092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.055069    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6c78f1-4137-4146-958b-726d999c64b5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs\" (UID: \"5b6c78f1-4137-4146-958b-726d999c64b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:20:55.090799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.088845    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rv9g\" (UniqueName: \"kubernetes.io/projected/5b6c78f1-4137-4146-958b-726d999c64b5-kube-api-access-7rv9g\") pod \"payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs\" (UID: \"5b6c78f1-4137-4146-958b-726d999c64b5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:20:55.127462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.127396    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 14:20:55.127462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.127453    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 14:20:55.220244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.220213    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:20:55.224532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.224507    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:20:55.607720 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:55.607376    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6c78f1_4137_4146_958b_726d999c64b5.slice/crio-18eaeb2cbd876a5d68169739eda46edc2be303c3f8bb95b468982f80b108ece7 WatchSource:0}: Error finding container 18eaeb2cbd876a5d68169739eda46edc2be303c3f8bb95b468982f80b108ece7: Status 404 returned error can't find the container with id 18eaeb2cbd876a5d68169739eda46edc2be303c3f8bb95b468982f80b108ece7
Nov 02 14:20:55.607918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.607744    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"]
Nov 02 14:20:55.630235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.630211    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"]
Nov 02 14:20:55.630574 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:55.630449    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac001bf_2ca9_4400_a244_f4404bf2351c.slice/crio-d4d06e5dc2fe7aa5064e546ed2071f813543df0f12d23fffa570df584d880a59 WatchSource:0}: Error finding container d4d06e5dc2fe7aa5064e546ed2071f813543df0f12d23fffa570df584d880a59: Status 404 returned error can't find the container with id d4d06e5dc2fe7aa5064e546ed2071f813543df0f12d23fffa570df584d880a59
Nov 02 14:20:55.706531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.706508    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"]
Nov 02 14:20:55.713152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.713052    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:55.724952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.724381    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"]
Nov 02 14:20:55.729831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.729811    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:20:55.732754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.732647    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:20:55.732754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.732727    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:20:55.768371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.768338    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3ec515-fwhm8\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:55.768585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.768564    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3ec515-fwhm8\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:55.768655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.768601    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz2gr\" (UniqueName: \"kubernetes.io/projected/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-kube-api-access-dz2gr\") pod \"maas-api-revoke-sub-fa3ec515-fwhm8\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:55.830892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.830868    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:20:55.897809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.878821    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3ec515-fwhm8\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:55.900934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.879638    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa3ec515-fwhm8\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:55.901445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.901418    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz2gr\" (UniqueName: \"kubernetes.io/projected/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-kube-api-access-dz2gr\") pod \"maas-api-revoke-sub-fa3ec515-fwhm8\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:55.902277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.902257    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3ec515-fwhm8\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:55.906197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.906146    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa3ec515-fwhm8\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:55.932452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:55.932431    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz2gr\" (UniqueName: \"kubernetes.io/projected/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-kube-api-access-dz2gr\") pod \"maas-api-revoke-sub-fa3ec515-fwhm8\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:56.044690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.044663    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:56.127924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.127884    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 14:20:56.128211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.127940    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 14:20:56.408201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.408173    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"]
Nov 02 14:20:56.409052 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:20:56.409023    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9c7dc6_e36f_435e_b95c_3beaa43781fb.slice/crio-4ba18bb307218c6b5dd88626c8a54f093ac9711f077d03df461fc4e8ebb2a315 WatchSource:0}: Error finding container 4ba18bb307218c6b5dd88626c8a54f093ac9711f077d03df461fc4e8ebb2a315: Status 404 returned error can't find the container with id 4ba18bb307218c6b5dd88626c8a54f093ac9711f077d03df461fc4e8ebb2a315
Nov 02 14:20:56.448952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.448923    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz" event={"ID":"4ac001bf-2ca9-4400-a244-f4404bf2351c","Type":"ContainerStarted","Data":"c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c"}
Nov 02 14:20:56.449129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.448978    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz" event={"ID":"4ac001bf-2ca9-4400-a244-f4404bf2351c","Type":"ContainerStarted","Data":"d4d06e5dc2fe7aa5064e546ed2071f813543df0f12d23fffa570df584d880a59"}
Nov 02 14:20:56.449129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.449012    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:20:56.450614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.450586    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8" event={"ID":"9f9c7dc6-e36f-435e-b95c-3beaa43781fb","Type":"ContainerStarted","Data":"4ba18bb307218c6b5dd88626c8a54f093ac9711f077d03df461fc4e8ebb2a315"}
Nov 02 14:20:56.452310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.452284    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs" event={"ID":"5b6c78f1-4137-4146-958b-726d999c64b5","Type":"ContainerStarted","Data":"96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b"}
Nov 02 14:20:56.452401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.452336    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs" event={"ID":"5b6c78f1-4137-4146-958b-726d999c64b5","Type":"ContainerStarted","Data":"18eaeb2cbd876a5d68169739eda46edc2be303c3f8bb95b468982f80b108ece7"}
Nov 02 14:20:56.452401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.452357    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:20:56.500720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.500653    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz" podStartSLOduration=2.500635552 podStartE2EDuration="2.500635552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:56.500126286 +0000 UTC m=+2247.251430724" watchObservedRunningTime="2026-11-02 14:20:56.500635552 +0000 UTC m=+2247.251939992"
Nov 02 14:20:56.535603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.535565    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:20:56.549065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:56.549013    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs" podStartSLOduration=2.54899998 podStartE2EDuration="2.54899998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:56.542961659 +0000 UTC m=+2247.294266096" watchObservedRunningTime="2026-11-02 14:20:56.54899998 +0000 UTC m=+2247.300304590"
Nov 02 14:20:57.127688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:57.127657    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Nov 02 14:20:57.127916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:57.127709    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Nov 02 14:20:57.461915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:57.461814    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8" event={"ID":"9f9c7dc6-e36f-435e-b95c-3beaa43781fb","Type":"ContainerStarted","Data":"34b024d5dfa0c4631c7049a8841c86a0c63cd45cdf862fd2ddeb314a3a7980d3"}
Nov 02 14:20:57.507273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:57.507214    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8" podStartSLOduration=2.5071985310000002 podStartE2EDuration="2.507198531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:20:57.505431364 +0000 UTC m=+2248.256735801" watchObservedRunningTime="2026-11-02 14:20:57.507198531 +0000 UTC m=+2248.258502993"
Nov 02 14:20:58.144171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:58.144081    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:20:58.146537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:58.146511    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:20:58.471259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:58.471172    5034 generic.go:356] "Generic (PLEG): container finished" podID="9f9c7dc6-e36f-435e-b95c-3beaa43781fb" containerID="34b024d5dfa0c4631c7049a8841c86a0c63cd45cdf862fd2ddeb314a3a7980d3" exitCode=0
Nov 02 14:20:58.471665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:58.471251    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8" event={"ID":"9f9c7dc6-e36f-435e-b95c-3beaa43781fb","Type":"ContainerDied","Data":"34b024d5dfa0c4631c7049a8841c86a0c63cd45cdf862fd2ddeb314a3a7980d3"}
Nov 02 14:20:59.323169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.323138    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:20:59.657507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.657478    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:20:59.793085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.793058    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-openshift-ca\") pod \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") "
Nov 02 14:20:59.793269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.793118    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-kube-api-access-dz2gr\" (UniqueName: \"kubernetes.io/projected/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-kube-api-access-dz2gr\") pod \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") "
Nov 02 14:20:59.793269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.793176    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-xks-ca\") pod \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\" (UID: \"9f9c7dc6-e36f-435e-b95c-3beaa43781fb\") "
Nov 02 14:20:59.793567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.793534    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-openshift-ca" pod "9f9c7dc6-e36f-435e-b95c-3beaa43781fb" (UID: "9f9c7dc6-e36f-435e-b95c-3beaa43781fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:20:59.793978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.793951    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:59.795585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.795559    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-kube-api-access-dz2gr" pod "9f9c7dc6-e36f-435e-b95c-3beaa43781fb" (UID: "9f9c7dc6-e36f-435e-b95c-3beaa43781fb"). InnerVolumeSpecName "kube-api-access-dz2gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:20:59.795864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.795829    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-xks-ca" pod "9f9c7dc6-e36f-435e-b95c-3beaa43781fb" (UID: "9f9c7dc6-e36f-435e-b95c-3beaa43781fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:20:59.900885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.898968    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz2gr\" (UniqueName: \"kubernetes.io/projected/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-kube-api-access-dz2gr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:20:59.900885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:20:59.899015    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7dc6-e36f-435e-b95c-3beaa43781fb-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:21:00.490408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:00.490380    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"
Nov 02 14:21:00.490588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:00.490410    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8" event={"ID":"9f9c7dc6-e36f-435e-b95c-3beaa43781fb","Type":"ContainerDied","Data":"4ba18bb307218c6b5dd88626c8a54f093ac9711f077d03df461fc4e8ebb2a315"}
Nov 02 14:21:00.490588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:00.490453    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ba18bb307218c6b5dd88626c8a54f093ac9711f077d03df461fc4e8ebb2a315"
Nov 02 14:21:07.465025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:07.464997    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:21:07.465415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:07.465056    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:21:17.760700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.760668    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"]
Nov 02 14:21:17.761445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.761427    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9f9c7dc6-e36f-435e-b95c-3beaa43781fb" containerName="revoke-keys"
Nov 02 14:21:17.761493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.761447    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f9c7dc6-e36f-435e-b95c-3beaa43781fb" containerName="revoke-keys"
Nov 02 14:21:17.761589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.761579    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f9c7dc6-e36f-435e-b95c-3beaa43781fb" containerName="revoke-keys"
Nov 02 14:21:17.765694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.765677    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:17.785912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.785885    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"]
Nov 02 14:21:17.793712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.793685    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-87efd3\""
Nov 02 14:21:17.824016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.823987    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"]
Nov 02 14:21:17.829186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.829165    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:17.841717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.841690    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb\" (UID: \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:17.841985 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.841964    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t96sd\" (UniqueName: \"kubernetes.io/projected/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-kube-api-access-t96sd\") pod \"payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb\" (UID: \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:17.849448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.849424    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"]
Nov 02 14:21:17.861099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.861075    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-87efd3-dockercfg-22gsg\""
Nov 02 14:21:18.015861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:17.970088    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hdv\" (UniqueName: \"kubernetes.io/projected/ccee94a9-7644-45ab-968a-c08d8793e67e-kube-api-access-c4hdv\") pod \"payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql\" (UID: \"ccee94a9-7644-45ab-968a-c08d8793e67e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:18.026804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.019397    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t96sd\" (UniqueName: \"kubernetes.io/projected/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-kube-api-access-t96sd\") pod \"payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb\" (UID: \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:18.028953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.028873    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb\" (UID: \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:18.030413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.030383    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee94a9-7644-45ab-968a-c08d8793e67e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql\" (UID: \"ccee94a9-7644-45ab-968a-c08d8793e67e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:18.030737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.030077    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb\" (UID: \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:18.087890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.084626    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t96sd\" (UniqueName: \"kubernetes.io/projected/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-kube-api-access-t96sd\") pod \"payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb\" (UID: \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:18.131700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.131664    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee94a9-7644-45ab-968a-c08d8793e67e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql\" (UID: \"ccee94a9-7644-45ab-968a-c08d8793e67e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:18.131871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.131740    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hdv\" (UniqueName: \"kubernetes.io/projected/ccee94a9-7644-45ab-968a-c08d8793e67e-kube-api-access-c4hdv\") pod \"payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql\" (UID: \"ccee94a9-7644-45ab-968a-c08d8793e67e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:18.132440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.132421    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee94a9-7644-45ab-968a-c08d8793e67e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql\" (UID: \"ccee94a9-7644-45ab-968a-c08d8793e67e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:18.157907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.157877    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hdv\" (UniqueName: \"kubernetes.io/projected/ccee94a9-7644-45ab-968a-c08d8793e67e-kube-api-access-c4hdv\") pod \"payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql\" (UID: \"ccee94a9-7644-45ab-968a-c08d8793e67e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:18.380430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.380339    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:18.426984 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:21:18.426950    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:21:18.446399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.446370    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:18.702307 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:21:18.702277    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc650161_5d67_41d9_b9b4_2e7c1bfaaeb9.slice/crio-ac9c385b37693c0483886e23630bbde52ee4758deeb855d2e8f8df7127bd59d4 WatchSource:0}: Error finding container ac9c385b37693c0483886e23630bbde52ee4758deeb855d2e8f8df7127bd59d4: Status 404 returned error can't find the container with id ac9c385b37693c0483886e23630bbde52ee4758deeb855d2e8f8df7127bd59d4
Nov 02 14:21:18.705025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.705000    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"]
Nov 02 14:21:18.772181 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:21:18.772143    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccee94a9_7644_45ab_968a_c08d8793e67e.slice/crio-06b996497dac25157aa60bec23768ddc0f1696063c35d99e1bb82de5cbfadfe5 WatchSource:0}: Error finding container 06b996497dac25157aa60bec23768ddc0f1696063c35d99e1bb82de5cbfadfe5: Status 404 returned error can't find the container with id 06b996497dac25157aa60bec23768ddc0f1696063c35d99e1bb82de5cbfadfe5
Nov 02 14:21:18.774259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:18.774235    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"]
Nov 02 14:21:19.683872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:19.683815    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb" event={"ID":"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9","Type":"ContainerStarted","Data":"8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd"}
Nov 02 14:21:19.684266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:19.684216    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb" event={"ID":"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9","Type":"ContainerStarted","Data":"ac9c385b37693c0483886e23630bbde52ee4758deeb855d2e8f8df7127bd59d4"}
Nov 02 14:21:19.684714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:19.684690    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:19.686296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:19.686271    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql" event={"ID":"ccee94a9-7644-45ab-968a-c08d8793e67e","Type":"ContainerStarted","Data":"3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650"}
Nov 02 14:21:19.686406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:19.686317    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql" event={"ID":"ccee94a9-7644-45ab-968a-c08d8793e67e","Type":"ContainerStarted","Data":"06b996497dac25157aa60bec23768ddc0f1696063c35d99e1bb82de5cbfadfe5"}
Nov 02 14:21:19.686461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:19.686411    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:19.731920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:19.731876    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb" podStartSLOduration=2.731862828 podStartE2EDuration="2.731862828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:19.726380626 +0000 UTC m=+2270.477685065" watchObservedRunningTime="2026-11-02 14:21:19.731862828 +0000 UTC m=+2270.483167266"
Nov 02 14:21:19.778155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:19.778101    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql" podStartSLOduration=2.778085652 podStartE2EDuration="2.778085652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:19.771019427 +0000 UTC m=+2270.522323865" watchObservedRunningTime="2026-11-02 14:21:19.778085652 +0000 UTC m=+2270.529390090"
Nov 02 14:21:30.202206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.202170    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"]
Nov 02 14:21:30.209482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.209458    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.234239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.234213    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"]
Nov 02 14:21:30.240544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.240511    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-0c2554-openshift-default-dockercfg-z2kgn\""
Nov 02 14:21:30.326863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.326829    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-socket\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.327047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.326897    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-credential-socket\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.327047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.327020    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c074b624-bef7-47b6-b3cc-e390df934572-istio-podinfo\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.327047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.327044    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-certs\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.327195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.327069    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-data\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.327195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.327084    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c074b624-bef7-47b6-b3cc-e390df934572-istiod-ca-cert\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.327310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.327292    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-envoy\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.327357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.327330    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsvf\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-kube-api-access-wwsvf\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.327418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.327361    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-istio-token\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.430312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.430268    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-credential-socket\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.431768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.431742    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c074b624-bef7-47b6-b3cc-e390df934572-istio-podinfo\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.433140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.433119    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-certs\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.433606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.433588    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-data\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.433743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.433727    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c074b624-bef7-47b6-b3cc-e390df934572-istiod-ca-cert\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.434002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.433983    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-envoy\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.434142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.434126    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsvf\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-kube-api-access-wwsvf\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.434278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.434263    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-istio-token\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.434517 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.434498    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-socket\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.434972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.434944    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c074b624-bef7-47b6-b3cc-e390df934572-istiod-ca-cert\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.435191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.431606    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-credential-socket\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.435751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.435730    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-socket\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.447938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.433539    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-certs\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.448066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.434130    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-data\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.460725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.460663    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c074b624-bef7-47b6-b3cc-e390df934572-istio-podinfo\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.463989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.463946    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-istio-token\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.488637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.473145    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-envoy\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.489639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.489582    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsvf\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-kube-api-access-wwsvf\") pod \"e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") " pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.536234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.535968    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:30.698577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.698547    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:21:30.886651 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:21:30.886603    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc074b624_bef7_47b6_b3cc_e390df934572.slice/crio-8b2deac215f35940c15262a066c29960ef9bc16f652e7bf26ce767d39f1ceea1 WatchSource:0}: Error finding container 8b2deac215f35940c15262a066c29960ef9bc16f652e7bf26ce767d39f1ceea1: Status 404 returned error can't find the container with id 8b2deac215f35940c15262a066c29960ef9bc16f652e7bf26ce767d39f1ceea1
Nov 02 14:21:30.888302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.888279    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"]
Nov 02 14:21:30.889717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.889682    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:30.889882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.889754    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:30.889882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:30.889818    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:31.706088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:31.706062    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:21:31.805878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:31.805838    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" event={"ID":"c074b624-bef7-47b6-b3cc-e390df934572","Type":"ContainerStarted","Data":"3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6"}
Nov 02 14:21:31.806073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:31.805909    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" event={"ID":"c074b624-bef7-47b6-b3cc-e390df934572","Type":"ContainerStarted","Data":"8b2deac215f35940c15262a066c29960ef9bc16f652e7bf26ce767d39f1ceea1"}
Nov 02 14:21:31.856936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:31.856887    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" podStartSLOduration=1.8568710579999999 podStartE2EDuration="1.856871058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:31.855346534 +0000 UTC m=+2282.606650972" watchObservedRunningTime="2026-11-02 14:21:31.856871058 +0000 UTC m=+2282.608175495"
Nov 02 14:21:32.542002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:32.541961    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:32.547547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:32.547525    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:32.815872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:32.815790    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:32.816806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:32.816752    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:21:36.131294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.131259    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"]
Nov 02 14:21:36.136281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.136258    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:36.163481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.163443    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"]
Nov 02 14:21:36.166164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.166136    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-0c2554\""
Nov 02 14:21:36.166466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.166443    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-0c2554-dockercfg-bswq7\""
Nov 02 14:21:36.212902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.212855    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"]
Nov 02 14:21:36.219023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.218971    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:36.228475 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.228438    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"]
Nov 02 14:21:36.237644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.236805    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462d4760-4801-4ef0-83c8-f4ea8623c37c-plugins-config-volume\") pod \"payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs\" (UID: \"462d4760-4801-4ef0-83c8-f4ea8623c37c\") " pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:36.237644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.237160    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdwq\" (UniqueName: \"kubernetes.io/projected/462d4760-4801-4ef0-83c8-f4ea8623c37c-kube-api-access-dvdwq\") pod \"payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs\" (UID: \"462d4760-4801-4ef0-83c8-f4ea8623c37c\") " pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:36.355806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.354056    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462d4760-4801-4ef0-83c8-f4ea8623c37c-plugins-config-volume\") pod \"payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs\" (UID: \"462d4760-4801-4ef0-83c8-f4ea8623c37c\") " pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:36.355806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.354138    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a97b65-d1cc-49c8-8816-133bbf7c8d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz\" (UID: \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:36.355806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.354337    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9lhq\" (UniqueName: \"kubernetes.io/projected/96a97b65-d1cc-49c8-8816-133bbf7c8d96-kube-api-access-x9lhq\") pod \"payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz\" (UID: \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:36.355806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.354589    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdwq\" (UniqueName: \"kubernetes.io/projected/462d4760-4801-4ef0-83c8-f4ea8623c37c-kube-api-access-dvdwq\") pod \"payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs\" (UID: \"462d4760-4801-4ef0-83c8-f4ea8623c37c\") " pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:36.400862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.399061    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462d4760-4801-4ef0-83c8-f4ea8623c37c-plugins-config-volume\") pod \"payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs\" (UID: \"462d4760-4801-4ef0-83c8-f4ea8623c37c\") " pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:36.407487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.405769    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdwq\" (UniqueName: \"kubernetes.io/projected/462d4760-4801-4ef0-83c8-f4ea8623c37c-kube-api-access-dvdwq\") pod \"payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs\" (UID: \"462d4760-4801-4ef0-83c8-f4ea8623c37c\") " pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:36.485418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.477626    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9lhq\" (UniqueName: \"kubernetes.io/projected/96a97b65-d1cc-49c8-8816-133bbf7c8d96-kube-api-access-x9lhq\") pod \"payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz\" (UID: \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:36.485418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.477978    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a97b65-d1cc-49c8-8816-133bbf7c8d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz\" (UID: \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:36.485418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.478959    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a97b65-d1cc-49c8-8816-133bbf7c8d96-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz\" (UID: \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:36.485418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.480846    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:36.512019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.511985    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9lhq\" (UniqueName: \"kubernetes.io/projected/96a97b65-d1cc-49c8-8816-133bbf7c8d96-kube-api-access-x9lhq\") pod \"payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz\" (UID: \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:36.578804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.577575    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:36.816032 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:21:36.815056    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462d4760_4801_4ef0_83c8_f4ea8623c37c.slice/crio-a7dd0f40faf78d059c3ee796d801e69a3b0e4a338ee6a034ed885f551b25bf62 WatchSource:0}: Error finding container a7dd0f40faf78d059c3ee796d801e69a3b0e4a338ee6a034ed885f551b25bf62: Status 404 returned error can't find the container with id a7dd0f40faf78d059c3ee796d801e69a3b0e4a338ee6a034ed885f551b25bf62
Nov 02 14:21:36.817603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.817578    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"]
Nov 02 14:21:36.851701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.851671    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs" event={"ID":"462d4760-4801-4ef0-83c8-f4ea8623c37c","Type":"ContainerStarted","Data":"a7dd0f40faf78d059c3ee796d801e69a3b0e4a338ee6a034ed885f551b25bf62"}
Nov 02 14:21:36.911485 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:21:36.911461    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a97b65_d1cc_49c8_8816_133bbf7c8d96.slice/crio-6502f32ff48e1a05e1021e50f71a3fc4bef1929d1c95b479c626e3b02686f9ff WatchSource:0}: Error finding container 6502f32ff48e1a05e1021e50f71a3fc4bef1929d1c95b479c626e3b02686f9ff: Status 404 returned error can't find the container with id 6502f32ff48e1a05e1021e50f71a3fc4bef1929d1c95b479c626e3b02686f9ff
Nov 02 14:21:36.914007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:36.913980    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"]
Nov 02 14:21:37.861354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:37.861322    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz" event={"ID":"96a97b65-d1cc-49c8-8816-133bbf7c8d96","Type":"ContainerStarted","Data":"dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88"}
Nov 02 14:21:37.861819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:37.861375    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz" event={"ID":"96a97b65-d1cc-49c8-8816-133bbf7c8d96","Type":"ContainerStarted","Data":"6502f32ff48e1a05e1021e50f71a3fc4bef1929d1c95b479c626e3b02686f9ff"}
Nov 02 14:21:37.861819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:37.861404    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:37.863162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:37.863135    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs" event={"ID":"462d4760-4801-4ef0-83c8-f4ea8623c37c","Type":"ContainerStarted","Data":"735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc"}
Nov 02 14:21:37.863265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:37.863211    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:37.909725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:37.909664    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz" podStartSLOduration=1.909646647 podStartE2EDuration="1.909646647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:37.906168747 +0000 UTC m=+2288.657473186" watchObservedRunningTime="2026-11-02 14:21:37.909646647 +0000 UTC m=+2288.660951085"
Nov 02 14:21:37.953313 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:37.953258    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs" podStartSLOduration=1.9532441120000001 podStartE2EDuration="1.953244112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:37.948361421 +0000 UTC m=+2288.699665861" watchObservedRunningTime="2026-11-02 14:21:37.953244112 +0000 UTC m=+2288.704548550"
Nov 02 14:21:47.121947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.118963    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"]
Nov 02 14:21:47.126789 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.126750    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:21:47.146798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.143754    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"]
Nov 02 14:21:47.241802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.239120    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ee3eca-dockercfg-l9sbc\""
Nov 02 14:21:47.266803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.265172    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c830edf9-caad-47fd-88aa-1c1e24e50790-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv\" (UID: \"c830edf9-caad-47fd-88aa-1c1e24e50790\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:21:47.266803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.265444    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9j58\" (UniqueName: \"kubernetes.io/projected/c830edf9-caad-47fd-88aa-1c1e24e50790-kube-api-access-s9j58\") pod \"payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv\" (UID: \"c830edf9-caad-47fd-88aa-1c1e24e50790\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:21:47.435374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.395246    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ee3eca\""
Nov 02 14:21:47.435374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.434726    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9j58\" (UniqueName: \"kubernetes.io/projected/c830edf9-caad-47fd-88aa-1c1e24e50790-kube-api-access-s9j58\") pod \"payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv\" (UID: \"c830edf9-caad-47fd-88aa-1c1e24e50790\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:21:47.435374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.434894    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c830edf9-caad-47fd-88aa-1c1e24e50790-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv\" (UID: \"c830edf9-caad-47fd-88aa-1c1e24e50790\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:21:47.464984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.457482    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c830edf9-caad-47fd-88aa-1c1e24e50790-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv\" (UID: \"c830edf9-caad-47fd-88aa-1c1e24e50790\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:21:47.527198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.527149    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9j58\" (UniqueName: \"kubernetes.io/projected/c830edf9-caad-47fd-88aa-1c1e24e50790-kube-api-access-s9j58\") pod \"payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv\" (UID: \"c830edf9-caad-47fd-88aa-1c1e24e50790\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:21:47.527382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.527219    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"]
Nov 02 14:21:47.542031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.541991    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"]
Nov 02 14:21:47.542184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.542104    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:21:47.638751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.638707    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhrlx\" (UniqueName: \"kubernetes.io/projected/fd7597dc-d18f-46e0-b4df-6242defcbbe6-kube-api-access-fhrlx\") pod \"payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp\" (UID: \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:21:47.638966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.638836    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7597dc-d18f-46e0-b4df-6242defcbbe6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp\" (UID: \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:21:47.740150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.740123    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhrlx\" (UniqueName: \"kubernetes.io/projected/fd7597dc-d18f-46e0-b4df-6242defcbbe6-kube-api-access-fhrlx\") pod \"payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp\" (UID: \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:21:47.740317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.740176    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7597dc-d18f-46e0-b4df-6242defcbbe6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp\" (UID: \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:21:47.740958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.740938    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7597dc-d18f-46e0-b4df-6242defcbbe6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp\" (UID: \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:21:47.745695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.745670    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:21:47.764936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.764908    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhrlx\" (UniqueName: \"kubernetes.io/projected/fd7597dc-d18f-46e0-b4df-6242defcbbe6-kube-api-access-fhrlx\") pod \"payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp\" (UID: \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:21:47.863124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:47.862749    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:21:48.123842 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:21:48.100944    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc830edf9_caad_47fd_88aa_1c1e24e50790.slice/crio-6b6f2e586606894258f452a13c273aabb1f3c5c61610ffdbad00a5e16fea5690 WatchSource:0}: Error finding container 6b6f2e586606894258f452a13c273aabb1f3c5c61610ffdbad00a5e16fea5690: Status 404 returned error can't find the container with id 6b6f2e586606894258f452a13c273aabb1f3c5c61610ffdbad00a5e16fea5690
Nov 02 14:21:48.128125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:48.128095    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"]
Nov 02 14:21:48.278212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:48.278141    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"]
Nov 02 14:21:48.279855 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:21:48.279817    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7597dc_d18f_46e0_b4df_6242defcbbe6.slice/crio-ecae0f2aef60126d58b3dc5f56ea302af7c80f1fa007a64fef5a5a0f929a7398 WatchSource:0}: Error finding container ecae0f2aef60126d58b3dc5f56ea302af7c80f1fa007a64fef5a5a0f929a7398: Status 404 returned error can't find the container with id ecae0f2aef60126d58b3dc5f56ea302af7c80f1fa007a64fef5a5a0f929a7398
Nov 02 14:21:48.425201 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:21:48.425160    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:21:48.885283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:48.885242    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:21:48.887673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:48.887645    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:21:49.085073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:49.085036    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv" event={"ID":"c830edf9-caad-47fd-88aa-1c1e24e50790","Type":"ContainerStarted","Data":"f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24"}
Nov 02 14:21:49.085073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:49.085100    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv" event={"ID":"c830edf9-caad-47fd-88aa-1c1e24e50790","Type":"ContainerStarted","Data":"6b6f2e586606894258f452a13c273aabb1f3c5c61610ffdbad00a5e16fea5690"}
Nov 02 14:21:49.085426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:49.085404    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:21:49.087818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:49.087793    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp" event={"ID":"fd7597dc-d18f-46e0-b4df-6242defcbbe6","Type":"ContainerStarted","Data":"284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3"}
Nov 02 14:21:49.087935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:49.087838    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp" event={"ID":"fd7597dc-d18f-46e0-b4df-6242defcbbe6","Type":"ContainerStarted","Data":"ecae0f2aef60126d58b3dc5f56ea302af7c80f1fa007a64fef5a5a0f929a7398"}
Nov 02 14:21:49.088614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:49.088590    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:21:49.155512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:49.155412    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv" podStartSLOduration=2.155392494 podStartE2EDuration="2.155392494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:49.148236414 +0000 UTC m=+2299.899540855" watchObservedRunningTime="2026-11-02 14:21:49.155392494 +0000 UTC m=+2299.906696933"
Nov 02 14:21:49.193639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:49.193577    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp" podStartSLOduration=2.193558304 podStartE2EDuration="2.193558304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:49.190238691 +0000 UTC m=+2299.941543129" watchObservedRunningTime="2026-11-02 14:21:49.193558304 +0000 UTC m=+2299.944862743"
Nov 02 14:21:53.135077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:53.135042    5034 generic.go:356] "Generic (PLEG): container finished" podID="af80e624-1231-4cfd-8024-64373c76b18d" containerID="db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca" exitCode=0
Nov 02 14:21:53.135491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:53.135114    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" event={"ID":"af80e624-1231-4cfd-8024-64373c76b18d","Type":"ContainerDied","Data":"db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca"}
Nov 02 14:21:53.135588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:53.135574    5034 scope.go:122] "RemoveContainer" containerID="db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca"
Nov 02 14:21:53.137273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:53.137245    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:53.137371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:53.137316    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:53.137371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:53.137349    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:54.033741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.033703    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"]
Nov 02 14:21:54.038861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.038823    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.058344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.058321    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2d13f4-openshift-default-dockercfg-24bpg\""
Nov 02 14:21:54.112072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.112028    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-token\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.112072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.112065    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7c5551-d472-445c-b35f-b938c5dab57c-istiod-ca-cert\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.112346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.112092    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-data\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.112346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.112257    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-credential-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.112346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.112338    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwvrx\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-kube-api-access-kwvrx\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.112521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.112433    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-podinfo\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.112521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.112473    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-certs\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.112633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.112577    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.112633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.112618    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-envoy\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.128888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.126814    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"]
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.220680    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-credential-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.220791    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwvrx\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-kube-api-access-kwvrx\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.220880    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-podinfo\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.220928    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-certs\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.220990    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.221033    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-envoy\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.221149    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-token\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.221173    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7c5551-d472-445c-b35f-b938c5dab57c-istiod-ca-cert\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.221202    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-data\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.221658    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-data\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.257813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.221911    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-credential-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.263826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.259076    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 02 14:21:54.263826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.259154    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 02 14:21:54.296901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.294805    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-certs\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.296901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.295147    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-socket\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.320479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.318052    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7c5551-d472-445c-b35f-b938c5dab57c-istiod-ca-cert\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.412125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.408686    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-token\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.414441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.414374    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwvrx\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-kube-api-access-kwvrx\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.414833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.414806    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-envoy\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.419848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.419798    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-podinfo\") pod \"e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") " pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.422849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.422822    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" event={"ID":"af80e624-1231-4cfd-8024-64373c76b18d","Type":"ContainerStarted","Data":"55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484"}
Nov 02 14:21:54.425230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.425168    5034 generic.go:356] "Generic (PLEG): container finished" podID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerID="f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60" exitCode=0
Nov 02 14:21:54.425335 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.425305    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" event={"ID":"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9","Type":"ContainerDied","Data":"f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60"}
Nov 02 14:21:54.425641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.425625    5034 scope.go:122] "RemoveContainer" containerID="f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60"
Nov 02 14:21:54.428839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.428807    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:54.428936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.428896    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:54.428991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.428940    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:54.656082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.655990    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:54.687830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.682842    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:21:54.702805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.701907    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 14:21:54.702805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:54.701969    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 14:21:55.046645 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:21:55.046608    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7c5551_d472_445c_b35f_b938c5dab57c.slice/crio-5ee697b79abf65571a8a8336fa21e383da139fec9ccb2bcb9630f6a4305acd41 WatchSource:0}: Error finding container 5ee697b79abf65571a8a8336fa21e383da139fec9ccb2bcb9630f6a4305acd41: Status 404 returned error can't find the container with id 5ee697b79abf65571a8a8336fa21e383da139fec9ccb2bcb9630f6a4305acd41
Nov 02 14:21:55.048845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.048810    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"]
Nov 02 14:21:55.049486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.049455    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:55.049574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.049519    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:55.049574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.049555    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:21:55.446299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.446189    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" event={"ID":"3f7c5551-d472-445c-b35f-b938c5dab57c","Type":"ContainerStarted","Data":"e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755"}
Nov 02 14:21:55.446299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.446247    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" event={"ID":"3f7c5551-d472-445c-b35f-b938c5dab57c","Type":"ContainerStarted","Data":"5ee697b79abf65571a8a8336fa21e383da139fec9ccb2bcb9630f6a4305acd41"}
Nov 02 14:21:55.449680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.449654    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" event={"ID":"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9","Type":"ContainerStarted","Data":"8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c"}
Nov 02 14:21:55.529816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.527326    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" podStartSLOduration=2.527301649 podStartE2EDuration="2.527301649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:21:55.504538031 +0000 UTC m=+2306.255842472" watchObservedRunningTime="2026-11-02 14:21:55.527301649 +0000 UTC m=+2306.278606088"
Nov 02 14:21:55.660927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.660888    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:55.686360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.686325    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 14:21:55.686658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.686627    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 14:21:55.941669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.941635    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 02 14:21:55.941889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.941691    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 02 14:21:55.941975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:55.941957    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:21:56.664081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:56.664049    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:56.688804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:56.677238    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 14:21:56.688804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:56.677306    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 14:21:56.941677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:56.941585    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 02 14:21:56.941984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:56.941949    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 02 14:21:57.475803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:57.475133    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:57.487802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:57.481015    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:21:57.706805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:57.697922    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 14:21:57.706805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:57.697991    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 14:21:57.942047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:57.942009    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 02 14:21:57.942281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:57.942085    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 02 14:21:58.234861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:58.232872    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:21:58.680857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:58.678078    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Nov 02 14:21:58.680857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:58.678161    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Nov 02 14:21:58.941961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:58.941853    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 02 14:21:58.941961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:58.941924    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 02 14:21:59.942171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:59.942128    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 02 14:21:59.942668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:21:59.942604    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 02 14:22:00.018827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:00.018793    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:22:00.334854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:00.334822    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:22:00.340543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:00.340162    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:22:00.941689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:00.941648    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Nov 02 14:22:00.941957 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:00.941718    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Nov 02 14:22:01.116904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:01.116869    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:22:01.948122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:01.948092    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:22:01.948563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:01.948540    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:22:01.949641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:01.949617    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:22:12.443836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.443770    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"]
Nov 02 14:22:12.452935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.452897    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:12.469961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.469934    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"]
Nov 02 14:22:12.475697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.475670    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a02c35\""
Nov 02 14:22:12.476059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.476039    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a02c35-dockercfg-fx5cn\""
Nov 02 14:22:12.488152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.488128    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867\" (UID: \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:12.488323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.488308    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvl7\" (UniqueName: \"kubernetes.io/projected/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-kube-api-access-kvvl7\") pod \"payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867\" (UID: \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:12.648379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.648322    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867\" (UID: \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:12.692277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.669009    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvl7\" (UniqueName: \"kubernetes.io/projected/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-kube-api-access-kvvl7\") pod \"payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867\" (UID: \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:12.693591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.693370    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867\" (UID: \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:12.721187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.718549    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvl7\" (UniqueName: \"kubernetes.io/projected/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-kube-api-access-kvvl7\") pod \"payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867\" (UID: \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:12.784453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:12.784418    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:13.101314 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:22:13.101281    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce3a86b_ddf4_4b91_a2fb_909ecb45c052.slice/crio-87e68b8bf622030b06a1738e44449bb6374468801d3d7c9b7d5e4d707c3b786b WatchSource:0}: Error finding container 87e68b8bf622030b06a1738e44449bb6374468801d3d7c9b7d5e4d707c3b786b: Status 404 returned error can't find the container with id 87e68b8bf622030b06a1738e44449bb6374468801d3d7c9b7d5e4d707c3b786b
Nov 02 14:22:13.102370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:13.102338    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"]
Nov 02 14:22:13.730651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:13.730617    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867" event={"ID":"bce3a86b-ddf4-4b91-a2fb-909ecb45c052","Type":"ContainerStarted","Data":"2959f62e1ef0dd231f1d9531192da91ce88d356921fde1c7dfd61653955dacf7"}
Nov 02 14:22:13.731134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:13.730673    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867" event={"ID":"bce3a86b-ddf4-4b91-a2fb-909ecb45c052","Type":"ContainerStarted","Data":"87e68b8bf622030b06a1738e44449bb6374468801d3d7c9b7d5e4d707c3b786b"}
Nov 02 14:22:13.731134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:13.730717    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:13.781033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:13.780978    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867" podStartSLOduration=1.780964978 podStartE2EDuration="1.780964978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:13.777748732 +0000 UTC m=+2324.529053171" watchObservedRunningTime="2026-11-02 14:22:13.780964978 +0000 UTC m=+2324.532269416"
Nov 02 14:22:18.415403 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:22:18.415365    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:22:24.744296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:24.744265    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:22:34.294798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.290445    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"]
Nov 02 14:22:34.307796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.300318    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.317506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.317482    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-bc34db-openshift-default-dockercfg-4g4f9\""
Nov 02 14:22:34.336520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.336472    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"]
Nov 02 14:22:34.443328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.443288    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-socket\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.443524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.443436    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-token\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.443524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.443515    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-envoy\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.443671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.443551    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istiod-ca-cert\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.443671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.443573    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-certs\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.443770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.443672    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mllv6\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-kube-api-access-mllv6\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.443770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.443696    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-podinfo\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.444133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.444076    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-data\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.444232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.444155    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-credential-socket\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.550917    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mllv6\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-kube-api-access-mllv6\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.550970    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-podinfo\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.551043    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-data\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.551119    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-credential-socket\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.551194    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-socket\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.551339    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-token\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.551498    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-envoy\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.551544    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istiod-ca-cert\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.551580    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-certs\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.552223    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-certs\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.554023    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-data\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.554278    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-credential-socket\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.554513    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-socket\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.556855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.556234    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istiod-ca-cert\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.590808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.590341    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-envoy\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.607867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.598456    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-podinfo\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.607867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.598865    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mllv6\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-kube-api-access-mllv6\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.607867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.602335    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-token\") pod \"e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") " pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:34.626808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:34.624058    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:35.004180 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:22:35.004142    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7650e8be_f936_48bf_9b6f_1d7674c46cbf.slice/crio-b768585907f261429bfa04743c55650b269f1e7f5db6bf55e7b627d12ece3561 WatchSource:0}: Error finding container b768585907f261429bfa04743c55650b269f1e7f5db6bf55e7b627d12ece3561: Status 404 returned error can't find the container with id b768585907f261429bfa04743c55650b269f1e7f5db6bf55e7b627d12ece3561
Nov 02 14:22:35.007128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:35.007103    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"]
Nov 02 14:22:35.007616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:35.007587    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:22:35.007663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:35.007647    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:22:35.007696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:35.007675    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:22:35.965665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:35.965608    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" event={"ID":"7650e8be-f936-48bf-9b6f-1d7674c46cbf","Type":"ContainerStarted","Data":"097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948"}
Nov 02 14:22:35.966161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:35.965674    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" event={"ID":"7650e8be-f936-48bf-9b6f-1d7674c46cbf","Type":"ContainerStarted","Data":"b768585907f261429bfa04743c55650b269f1e7f5db6bf55e7b627d12ece3561"}
Nov 02 14:22:36.042219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:36.042167    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" podStartSLOduration=2.04215288 podStartE2EDuration="2.04215288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:36.027596387 +0000 UTC m=+2346.778900825" watchObservedRunningTime="2026-11-02 14:22:36.04215288 +0000 UTC m=+2346.793457365"
Nov 02 14:22:36.634802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:36.626089    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:36.663967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:36.663931    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:36.975973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:36.975881    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:36.976861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:36.976840    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:22:37.986613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:37.986578    5034 generic.go:356] "Generic (PLEG): container finished" podID="c074b624-bef7-47b6-b3cc-e390df934572" containerID="3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6" exitCode=0
Nov 02 14:22:37.987199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:37.986656    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" event={"ID":"c074b624-bef7-47b6-b3cc-e390df934572","Type":"ContainerDied","Data":"3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6"}
Nov 02 14:22:37.987340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:37.987321    5034 scope.go:122] "RemoveContainer" containerID="3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6"
Nov 02 14:22:37.989125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:37.989096    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:22:37.989225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:37.989164    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:22:37.989225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:37.989216    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:22:39.009046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:39.008886    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" event={"ID":"c074b624-bef7-47b6-b3cc-e390df934572","Type":"ContainerStarted","Data":"fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf"}
Nov 02 14:22:39.541809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:39.541747    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:22:39.542018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:39.542000    5034 patch_prober.go:29] interesting pod/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 02 14:22:39.542068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:39.542041    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 02 14:22:40.320388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.320350    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"]
Nov 02 14:22:40.325704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.325680    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:22:40.347040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.347012    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"]
Nov 02 14:22:40.353184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.353161    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bc34db\""
Nov 02 14:22:40.353360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.353291    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bc34db-dockercfg-p9nt5\""
Nov 02 14:22:40.418571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.418145    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grp8\" (UniqueName: \"kubernetes.io/projected/f28a0afb-2741-4a34-ab83-ff5af083c1a8-kube-api-access-6grp8\") pod \"payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h\" (UID: \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:22:40.418571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.418314    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28a0afb-2741-4a34-ab83-ff5af083c1a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h\" (UID: \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:22:40.541805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.525607    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grp8\" (UniqueName: \"kubernetes.io/projected/f28a0afb-2741-4a34-ab83-ff5af083c1a8-kube-api-access-6grp8\") pod \"payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h\" (UID: \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:22:40.604859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.573389    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28a0afb-2741-4a34-ab83-ff5af083c1a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h\" (UID: \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:22:40.654804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.574712    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28a0afb-2741-4a34-ab83-ff5af083c1a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h\" (UID: \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:22:40.655222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.655189    5034 patch_prober.go:29] interesting pod/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 02 14:22:40.655394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.655369    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 02 14:22:40.723687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.723608    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grp8\" (UniqueName: \"kubernetes.io/projected/f28a0afb-2741-4a34-ab83-ff5af083c1a8-kube-api-access-6grp8\") pod \"payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h\" (UID: \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:22:40.940754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:40.940666    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:22:41.309195 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:22:41.309150    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28a0afb_2741_4a34_ab83_ff5af083c1a8.slice/crio-270513a627d07e9d8e1ddd0d58eefabfdf937c61a31edf58011e1e2a00899450 WatchSource:0}: Error finding container 270513a627d07e9d8e1ddd0d58eefabfdf937c61a31edf58011e1e2a00899450: Status 404 returned error can't find the container with id 270513a627d07e9d8e1ddd0d58eefabfdf937c61a31edf58011e1e2a00899450
Nov 02 14:22:41.311709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:41.311680    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"]
Nov 02 14:22:41.541544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:41.541508    5034 patch_prober.go:29] interesting pod/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 02 14:22:41.541990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:41.541562    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 02 14:22:42.041001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:42.040973    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h" event={"ID":"f28a0afb-2741-4a34-ab83-ff5af083c1a8","Type":"ContainerStarted","Data":"84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2"}
Nov 02 14:22:42.041267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:42.041221    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:22:42.041267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:42.041245    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h" event={"ID":"f28a0afb-2741-4a34-ab83-ff5af083c1a8","Type":"ContainerStarted","Data":"270513a627d07e9d8e1ddd0d58eefabfdf937c61a31edf58011e1e2a00899450"}
Nov 02 14:22:42.108830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:42.107727    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h" podStartSLOduration=2.107708993 podStartE2EDuration="2.107708993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:22:42.085131804 +0000 UTC m=+2352.836436244" watchObservedRunningTime="2026-11-02 14:22:42.107708993 +0000 UTC m=+2352.859013434"
Nov 02 14:22:42.541474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:42.541442    5034 patch_prober.go:29] interesting pod/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 02 14:22:42.541659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:42.541490    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 02 14:22:43.541989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:43.541953    5034 patch_prober.go:29] interesting pod/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused" start-of-body=
Nov 02 14:22:43.542405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:43.542004    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.9:15021/healthz/ready\": dial tcp 10.132.1.9:15021: connect: connection refused"
Nov 02 14:22:44.644106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:44.644070    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:22:44.644840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:44.644816    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:22:44.647483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:44.647446    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:22:48.415121 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:22:48.415082    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:22:53.052404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:22:53.052375    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:23:11.464276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.464198    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"]
Nov 02 14:23:11.470466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.470434    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.492889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.492482    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"]
Nov 02 14:23:11.498183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.498154    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ee3eca\""
Nov 02 14:23:11.498430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.498412    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"kube-root-ca.crt\""
Nov 02 14:23:11.498509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.498479    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"openshift-service-ca.crt\""
Nov 02 14:23:11.498574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.498523    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"routing-overlay\""
Nov 02 14:23:11.502290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.502267    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-e2e--6a9d33b0-dockercfg-b6wf9\""
Nov 02 14:23:11.502473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.502460    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-821b4dcc9b92\""
Nov 02 14:23:11.548867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.548841    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthwp\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-kube-api-access-mthwp\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.549037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.548880    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.549037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.548995    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.549143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.549037    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.652799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.652610    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthwp\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-kube-api-access-mthwp\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.652799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.652678    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.652799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.652770    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.653140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.652851    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.662199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.653867    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.672796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.668571    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.683879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.679061    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.693497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.691680    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthwp\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-kube-api-access-mthwp\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:11.787894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:11.787856    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:12.100911 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:23:12.100877    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521cf372_2048_44b9_9fe3_67e3d9f42664.slice/crio-410a087a49b4aa835e2e37148aa8091f987b259b591a5cd22a978d97de204054 WatchSource:0}: Error finding container 410a087a49b4aa835e2e37148aa8091f987b259b591a5cd22a978d97de204054: Status 404 returned error can't find the container with id 410a087a49b4aa835e2e37148aa8091f987b259b591a5cd22a978d97de204054
Nov 02 14:23:12.101440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:12.101414    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"]
Nov 02 14:23:12.440419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:12.440330    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb" event={"ID":"521cf372-2048-44b9-9fe3-67e3d9f42664","Type":"ContainerStarted","Data":"07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a"}
Nov 02 14:23:12.440419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:12.440391    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb" event={"ID":"521cf372-2048-44b9-9fe3-67e3d9f42664","Type":"ContainerStarted","Data":"410a087a49b4aa835e2e37148aa8091f987b259b591a5cd22a978d97de204054"}
Nov 02 14:23:12.440662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:12.440424    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:12.490398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:12.490336    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb" podStartSLOduration=1.490317787 podStartE2EDuration="1.490317787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:12.484662709 +0000 UTC m=+2383.235967147" watchObservedRunningTime="2026-11-02 14:23:12.490317787 +0000 UTC m=+2383.241622226"
Nov 02 14:23:18.425267 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:23:18.425231    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:23:23.453734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:23.453656    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:23:27.080451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.080365    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"]
Nov 02 14:23:27.081030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.080625    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy" containerID="cri-o://cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3" gracePeriod=30
Nov 02 14:23:27.476601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.476565    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 02 14:23:27.476828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.476636    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 02 14:23:27.676480 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.676442    5034 generic.go:356] "Generic (PLEG): container finished" podID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerID="e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755" exitCode=0
Nov 02 14:23:27.676662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.676510    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" event={"ID":"3f7c5551-d472-445c-b35f-b938c5dab57c","Type":"ContainerDied","Data":"e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755"}
Nov 02 14:23:27.676956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.676934    5034 scope.go:122] "RemoveContainer" containerID="e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755"
Nov 02 14:23:27.678725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.678699    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:23:27.678838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.678769    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:23:27.678838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:27.678822    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:23:28.689043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:28.689009    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" event={"ID":"3f7c5551-d472-445c-b35f-b938c5dab57c","Type":"ContainerStarted","Data":"945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09"}
Nov 02 14:23:29.656649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:29.656615    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:23:29.656868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:29.656812    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 02 14:23:29.656957 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:29.656877    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 02 14:23:30.526170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:30.526139    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:23:30.527049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:30.527013    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:23:30.657084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:30.657050    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 02 14:23:30.657265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:30.657115    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 02 14:23:31.656675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:31.656638    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 02 14:23:31.657130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:31.656692    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 02 14:23:32.467451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.467428    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:23:32.508433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508395    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-socket\") pod \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") "
Nov 02 14:23:32.508605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508449    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-data\") pod \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") "
Nov 02 14:23:32.508605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508481    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-kube-api-access-t7kl9\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-kube-api-access-t7kl9\") pod \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") "
Nov 02 14:23:32.508744 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508635    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-certs\") pod \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") "
Nov 02 14:23:32.508744 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508732    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-socket" pod "5d46d575-e8a7-4171-8a7a-6ecd59b16728" (UID: "5d46d575-e8a7-4171-8a7a-6ecd59b16728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:32.508924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508765    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-podinfo\") pod \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") "
Nov 02 14:23:32.508924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508846    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-data" pod "5d46d575-e8a7-4171-8a7a-6ecd59b16728" (UID: "5d46d575-e8a7-4171-8a7a-6ecd59b16728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:32.508924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508853    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-envoy\") pod \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") "
Nov 02 14:23:32.509089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508976    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-credential-socket\") pod \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") "
Nov 02 14:23:32.509089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.508997    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-certs" pod "5d46d575-e8a7-4171-8a7a-6ecd59b16728" (UID: "5d46d575-e8a7-4171-8a7a-6ecd59b16728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:32.509089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.509033    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istiod-ca-cert\") pod \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") "
Nov 02 14:23:32.509263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.509132    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-token\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-token\") pod \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\" (UID: \"5d46d575-e8a7-4171-8a7a-6ecd59b16728\") "
Nov 02 14:23:32.509263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.509214    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-credential-socket" pod "5d46d575-e8a7-4171-8a7a-6ecd59b16728" (UID: "5d46d575-e8a7-4171-8a7a-6ecd59b16728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:32.510062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.509492    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istiod-ca-cert" pod "5d46d575-e8a7-4171-8a7a-6ecd59b16728" (UID: "5d46d575-e8a7-4171-8a7a-6ecd59b16728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:23:32.510062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.509942    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:32.510062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.509964    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:32.510062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.509979    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:32.510062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.509990    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:32.510062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.510037    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:32.511502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.511468    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-envoy" pod "5d46d575-e8a7-4171-8a7a-6ecd59b16728" (UID: "5d46d575-e8a7-4171-8a7a-6ecd59b16728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:32.512069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.512039    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-podinfo" pod "5d46d575-e8a7-4171-8a7a-6ecd59b16728" (UID: "5d46d575-e8a7-4171-8a7a-6ecd59b16728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:23:32.512174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.512096    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-kube-api-access-t7kl9" pod "5d46d575-e8a7-4171-8a7a-6ecd59b16728" (UID: "5d46d575-e8a7-4171-8a7a-6ecd59b16728"). InnerVolumeSpecName "kube-api-access-t7kl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:32.512174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.512127    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-token" pod "5d46d575-e8a7-4171-8a7a-6ecd59b16728" (UID: "5d46d575-e8a7-4171-8a7a-6ecd59b16728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:32.612423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.612381    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:32.612423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.612420    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:32.612698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.612433    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:32.612698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.612448    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7kl9\" (UniqueName: \"kubernetes.io/projected/5d46d575-e8a7-4171-8a7a-6ecd59b16728-kube-api-access-t7kl9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:32.656669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.656635    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 02 14:23:32.656946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.656693    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 02 14:23:32.741088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.741058    5034 generic.go:356] "Generic (PLEG): container finished" podID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerID="cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3" exitCode=0
Nov 02 14:23:32.741253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.741112    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"
Nov 02 14:23:32.741253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.741142    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" event={"ID":"5d46d575-e8a7-4171-8a7a-6ecd59b16728","Type":"ContainerDied","Data":"cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3"}
Nov 02 14:23:32.741253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.741202    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt" event={"ID":"5d46d575-e8a7-4171-8a7a-6ecd59b16728","Type":"ContainerDied","Data":"bc9f9ffe293d034bb372008a52c5417883380ee1aa18fa4236e435aa8f2e6fa9"}
Nov 02 14:23:32.741253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.741232    5034 scope.go:122] "RemoveContainer" containerID="cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3"
Nov 02 14:23:32.753892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.753870    5034 scope.go:122] "RemoveContainer" containerID="48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c"
Nov 02 14:23:32.764071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.764050    5034 scope.go:122] "RemoveContainer" containerID="cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3"
Nov 02 14:23:32.764332 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:23:32.764314    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3\": container with ID starting with cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3 not found: ID does not exist" containerID="cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3"
Nov 02 14:23:32.764384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.764342    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3"} err="failed to get container status \"cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3\": rpc error: code = NotFound desc = could not find container \"cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3\": container with ID starting with cc2c653e1cd553b2ac8cf196b43eaf09adc95ee9e17812e78c82ae118a9f65a3 not found: ID does not exist"
Nov 02 14:23:32.764384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.764359    5034 scope.go:122] "RemoveContainer" containerID="48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c"
Nov 02 14:23:32.764591 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:23:32.764570    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c\": container with ID starting with 48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c not found: ID does not exist" containerID="48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c"
Nov 02 14:23:32.764629 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.764598    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c"} err="failed to get container status \"48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c\": rpc error: code = NotFound desc = could not find container \"48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c\": container with ID starting with 48315ff9cd47cdf3b4238f11f55afa3b41b9abcf5d216ffb34c1bafa3bf31a9c not found: ID does not exist"
Nov 02 14:23:32.824862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.809045    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"]
Nov 02 14:23:32.825063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:32.824930    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-mwlmt"]
Nov 02 14:23:33.411960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.411925    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" path="/var/lib/kubelet/pods/5d46d575-e8a7-4171-8a7a-6ecd59b16728/volumes"
Nov 02 14:23:33.667360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.667269    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body=
Nov 02 14:23:33.667988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.667954    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused"
Nov 02 14:23:33.866841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.866803    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"]
Nov 02 14:23:33.880455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.868525    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy"
Nov 02 14:23:33.881594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.881565    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy"
Nov 02 14:23:33.887873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.887845    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy"
Nov 02 14:23:33.888042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.888028    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy"
Nov 02 14:23:33.888419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.888403    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy"
Nov 02 14:23:33.912119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.912091    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d46d575-e8a7-4171-8a7a-6ecd59b16728" containerName="istio-proxy"
Nov 02 14:23:33.994854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:33.970899    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.074274    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-socket\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.074383    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-certs\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.074445    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf6bk\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-kube-api-access-lf6bk\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.074519    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-envoy\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.074581    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-credential-socket\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.074694    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-token\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.074762    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbee88c-756f-40fe-a43f-3da83ea117a7-istiod-ca-cert\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.074826    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-data\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.074934    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-podinfo\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.081800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.079083    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-605541-openshift-default-dockercfg-225kj\""
Nov 02 14:23:34.086196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.086028    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"]
Nov 02 14:23:34.204801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.184657    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-credential-socket\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.205158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.205132    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-token\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.205329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.205315    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbee88c-756f-40fe-a43f-3da83ea117a7-istiod-ca-cert\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.205458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.205442    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-data\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.205630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.205618    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-podinfo\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.205754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.205742    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-socket\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.205930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.205915    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-certs\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.206076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.206062    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf6bk\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-kube-api-access-lf6bk\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.206224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.206211    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-envoy\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.206387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.185194    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-credential-socket\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.207417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.207392    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbee88c-756f-40fe-a43f-3da83ea117a7-istiod-ca-cert\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.207804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.207771    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-data\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.209106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.209082    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-certs\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.210138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.210078    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-socket\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.213372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.213325    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-podinfo\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.213807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.213767    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-envoy\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.233889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.233860    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf6bk\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-kube-api-access-lf6bk\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.234999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.234976    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-token\") pod \"e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") " pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.383637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.382816    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:34.701489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.701376    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:23:34.703025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.703001    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:23:34.703720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.703704    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:23:34.742226 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:23:34.742200    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbee88c_756f_40fe_a43f_3da83ea117a7.slice/crio-d254ec49d99c9f36e92d6892bf5a45b6b396bda3a40ae5481657c812aa1b8576 WatchSource:0}: Error finding container d254ec49d99c9f36e92d6892bf5a45b6b396bda3a40ae5481657c812aa1b8576: Status 404 returned error can't find the container with id d254ec49d99c9f36e92d6892bf5a45b6b396bda3a40ae5481657c812aa1b8576
Nov 02 14:23:34.745323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.745298    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"]
Nov 02 14:23:34.745457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.745427    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:23:34.745500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.745490    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:23:34.745541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.745521    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:23:34.799642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:34.799609    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" event={"ID":"edbee88c-756f-40fe-a43f-3da83ea117a7","Type":"ContainerStarted","Data":"d254ec49d99c9f36e92d6892bf5a45b6b396bda3a40ae5481657c812aa1b8576"}
Nov 02 14:23:35.917803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:35.915932    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" event={"ID":"edbee88c-756f-40fe-a43f-3da83ea117a7","Type":"ContainerStarted","Data":"2e4f925401a8fa0f3a41f51c03d79d7539dde864271740a4e48333eec76e6773"}
Nov 02 14:23:35.973212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:35.973146    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" podStartSLOduration=2.973128356 podStartE2EDuration="2.973128356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:35.967258456 +0000 UTC m=+2406.718562888" watchObservedRunningTime="2026-11-02 14:23:35.973128356 +0000 UTC m=+2406.724432798"
Nov 02 14:23:36.383586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:36.383543    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:36.390460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:36.390427    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:36.918879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:36.918698    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:36.930977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:36.930901    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:23:38.810818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:38.810734    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"]
Nov 02 14:23:38.811347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:38.811001    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy" containerID="cri-o://f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9" gracePeriod=30
Nov 02 14:23:40.655103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.655067    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"]
Nov 02 14:23:40.659728 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.659696    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:40.744983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.734280    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-605541-dockercfg-s249f\""
Nov 02 14:23:40.805935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.805895    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-762j4\" (UniqueName: \"kubernetes.io/projected/0fca14ad-ef87-4215-ae1d-2e760a55236b-kube-api-access-762j4\") pod \"payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx\" (UID: \"0fca14ad-ef87-4215-ae1d-2e760a55236b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:40.822131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.822099    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fca14ad-ef87-4215-ae1d-2e760a55236b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx\" (UID: \"0fca14ad-ef87-4215-ae1d-2e760a55236b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:40.927260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.927130    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-605541\""
Nov 02 14:23:40.967950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.941153    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fca14ad-ef87-4215-ae1d-2e760a55236b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx\" (UID: \"0fca14ad-ef87-4215-ae1d-2e760a55236b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:40.968720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.968677    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-762j4\" (UniqueName: \"kubernetes.io/projected/0fca14ad-ef87-4215-ae1d-2e760a55236b-kube-api-access-762j4\") pod \"payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx\" (UID: \"0fca14ad-ef87-4215-ae1d-2e760a55236b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:40.987204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.971078    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fca14ad-ef87-4215-ae1d-2e760a55236b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx\" (UID: \"0fca14ad-ef87-4215-ae1d-2e760a55236b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:40.987204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:40.985734    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"]
Nov 02 14:23:41.039141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:41.039105    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-762j4\" (UniqueName: \"kubernetes.io/projected/0fca14ad-ef87-4215-ae1d-2e760a55236b-kube-api-access-762j4\") pod \"payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx\" (UID: \"0fca14ad-ef87-4215-ae1d-2e760a55236b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:41.228625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:41.228535    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:41.657821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:41.656359    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"]
Nov 02 14:23:42.151633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:42.150031    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx" event={"ID":"0fca14ad-ef87-4215-ae1d-2e760a55236b","Type":"ContainerStarted","Data":"d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078"}
Nov 02 14:23:42.151633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:42.150115    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx" event={"ID":"0fca14ad-ef87-4215-ae1d-2e760a55236b","Type":"ContainerStarted","Data":"fb5559c5dc4f3a79e9b76c93d622e5b5ece01b865266df7d10bc7160e8100b49"}
Nov 02 14:23:42.151633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:42.151297    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:42.214125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:42.214046    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx" podStartSLOduration=2.214011011 podStartE2EDuration="2.214011011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:23:42.207271492 +0000 UTC m=+2412.958575935" watchObservedRunningTime="2026-11-02 14:23:42.214011011 +0000 UTC m=+2412.965315449"
Nov 02 14:23:44.137099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.137073    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:23:44.172030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.171992    5034 generic.go:356] "Generic (PLEG): container finished" podID="b48deb00-e178-4318-a592-c53c7899bddf" containerID="f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9" exitCode=0
Nov 02 14:23:44.172925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.172902    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"
Nov 02 14:23:44.173084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.173066    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" event={"ID":"b48deb00-e178-4318-a592-c53c7899bddf","Type":"ContainerDied","Data":"f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9"}
Nov 02 14:23:44.173172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.173113    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q" event={"ID":"b48deb00-e178-4318-a592-c53c7899bddf","Type":"ContainerDied","Data":"0fab8dc7589b89fb6e9c98e058cdc602a1da8d6b403130bf0a8a97ed2a8c640a"}
Nov 02 14:23:44.173172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.173143    5034 scope.go:122] "RemoveContainer" containerID="f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9"
Nov 02 14:23:44.187484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.187460    5034 scope.go:122] "RemoveContainer" containerID="f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9"
Nov 02 14:23:44.187943 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:23:44.187917    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9\": container with ID starting with f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9 not found: ID does not exist" containerID="f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9"
Nov 02 14:23:44.188057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.187954    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9"} err="failed to get container status \"f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9\": rpc error: code = NotFound desc = could not find container \"f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9\": container with ID starting with f815222147dc277c3cd0c0736bb45dc4a257906d0b012df7fc87b40cbb8364e9 not found: ID does not exist"
Nov 02 14:23:44.271683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.271594    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-certs\") pod \"b48deb00-e178-4318-a592-c53c7899bddf\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") "
Nov 02 14:23:44.271866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.271767    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-credential-socket\") pod \"b48deb00-e178-4318-a592-c53c7899bddf\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") "
Nov 02 14:23:44.271925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.271903    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-data\") pod \"b48deb00-e178-4318-a592-c53c7899bddf\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") "
Nov 02 14:23:44.271994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.271967    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-certs" pod "b48deb00-e178-4318-a592-c53c7899bddf" (UID: "b48deb00-e178-4318-a592-c53c7899bddf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:44.272125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.272082    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-credential-socket" pod "b48deb00-e178-4318-a592-c53c7899bddf" (UID: "b48deb00-e178-4318-a592-c53c7899bddf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:44.272173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.272154    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-data" pod "b48deb00-e178-4318-a592-c53c7899bddf" (UID: "b48deb00-e178-4318-a592-c53c7899bddf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:44.272288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.272268    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-envoy\") pod \"b48deb00-e178-4318-a592-c53c7899bddf\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") "
Nov 02 14:23:44.272356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.272331    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b48deb00-e178-4318-a592-c53c7899bddf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48deb00-e178-4318-a592-c53c7899bddf-istio-podinfo\") pod \"b48deb00-e178-4318-a592-c53c7899bddf\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") "
Nov 02 14:23:44.272432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.272417    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-socket\") pod \"b48deb00-e178-4318-a592-c53c7899bddf\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") "
Nov 02 14:23:44.272489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.272461    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-istio-token\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-istio-token\") pod \"b48deb00-e178-4318-a592-c53c7899bddf\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") "
Nov 02 14:23:44.272620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.272603    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48deb00-e178-4318-a592-c53c7899bddf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48deb00-e178-4318-a592-c53c7899bddf-istiod-ca-cert\") pod \"b48deb00-e178-4318-a592-c53c7899bddf\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") "
Nov 02 14:23:44.272679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.272657    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-kube-api-access-jvmz8\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-kube-api-access-jvmz8\") pod \"b48deb00-e178-4318-a592-c53c7899bddf\" (UID: \"b48deb00-e178-4318-a592-c53c7899bddf\") "
Nov 02 14:23:44.273014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.272988    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-socket" pod "b48deb00-e178-4318-a592-c53c7899bddf" (UID: "b48deb00-e178-4318-a592-c53c7899bddf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:44.273014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.273004    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48deb00-e178-4318-a592-c53c7899bddf-istiod-ca-cert" pod "b48deb00-e178-4318-a592-c53c7899bddf" (UID: "b48deb00-e178-4318-a592-c53c7899bddf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:23:44.273956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.273930    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b48deb00-e178-4318-a592-c53c7899bddf-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:44.274081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.273959    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:44.274081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.273974    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:44.274081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.273989    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:44.274081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.274002    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:44.276032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.276002    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b48deb00-e178-4318-a592-c53c7899bddf-istio-podinfo" pod "b48deb00-e178-4318-a592-c53c7899bddf" (UID: "b48deb00-e178-4318-a592-c53c7899bddf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:23:44.276032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.276018    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-istio-token" pod "b48deb00-e178-4318-a592-c53c7899bddf" (UID: "b48deb00-e178-4318-a592-c53c7899bddf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:44.276213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.276126    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-kube-api-access-jvmz8" pod "b48deb00-e178-4318-a592-c53c7899bddf" (UID: "b48deb00-e178-4318-a592-c53c7899bddf"). InnerVolumeSpecName "kube-api-access-jvmz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:23:44.276951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.276922    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-envoy" pod "b48deb00-e178-4318-a592-c53c7899bddf" (UID: "b48deb00-e178-4318-a592-c53c7899bddf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:23:44.374742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.374708    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvmz8\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-kube-api-access-jvmz8\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:44.374742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.374740    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b48deb00-e178-4318-a592-c53c7899bddf-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:44.374742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.374750    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b48deb00-e178-4318-a592-c53c7899bddf-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:44.375057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.374758    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b48deb00-e178-4318-a592-c53c7899bddf-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:23:44.528924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.528856    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"]
Nov 02 14:23:44.538615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:44.538588    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-brz6q"]
Nov 02 14:23:45.399849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:45.399807    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48deb00-e178-4318-a592-c53c7899bddf" path="/var/lib/kubelet/pods/b48deb00-e178-4318-a592-c53c7899bddf/volumes"
Nov 02 14:23:48.216909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:48.216820    5034 generic.go:356] "Generic (PLEG): container finished" podID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerID="097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948" exitCode=0
Nov 02 14:23:48.216909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:48.216887    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" event={"ID":"7650e8be-f936-48bf-9b6f-1d7674c46cbf","Type":"ContainerDied","Data":"097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948"}
Nov 02 14:23:48.217366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:48.217334    5034 scope.go:122] "RemoveContainer" containerID="097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948"
Nov 02 14:23:48.219193 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:48.219163    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:23:48.219327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:48.219248    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:23:48.219327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:48.219297    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:23:48.420941 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:23:48.420902    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:23:49.227857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:49.227822    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" event={"ID":"7650e8be-f936-48bf-9b6f-1d7674c46cbf","Type":"ContainerStarted","Data":"319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf"}
Nov 02 14:23:49.625596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:49.625566    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:23:49.626630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:49.626599    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body=
Nov 02 14:23:49.626808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:49.626656    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused"
Nov 02 14:23:50.625356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:50.625325    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body=
Nov 02 14:23:50.625751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:50.625373    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused"
Nov 02 14:23:51.625447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:51.625416    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body=
Nov 02 14:23:51.625891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:51.625471    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused"
Nov 02 14:23:51.993798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:51.985856    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:23:52.625190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:52.625156    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body=
Nov 02 14:23:52.625379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:52.625220    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused"
Nov 02 14:23:53.625293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:53.625263    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body=
Nov 02 14:23:53.625713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:53.625312    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused"
Nov 02 14:23:54.188107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:54.188075    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:23:54.630185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:54.630157    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:23:54.631461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:54.631437    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:23:55.482840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:55.482742    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"]
Nov 02 14:23:55.507003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:23:55.506970    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"]
Nov 02 14:24:00.483723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:00.483676    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs" podUID="462d4760-4801-4ef0-83c8-f4ea8623c37c" containerName="payload-processing" containerID="cri-o://735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc" gracePeriod=25
Nov 02 14:24:00.508081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:00.508005    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz" podUID="96a97b65-d1cc-49c8-8816-133bbf7c8d96" containerName="payload-pre-processing" containerID="cri-o://dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88" gracePeriod=25
Nov 02 14:24:00.830606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:00.830576    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:24:00.857861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:00.857842    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:24:01.027431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.027398    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a97b65-d1cc-49c8-8816-133bbf7c8d96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a97b65-d1cc-49c8-8816-133bbf7c8d96-plugins-config-volume\") pod \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\" (UID: \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\") "
Nov 02 14:24:01.027614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.027571    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462d4760-4801-4ef0-83c8-f4ea8623c37c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462d4760-4801-4ef0-83c8-f4ea8623c37c-plugins-config-volume\") pod \"462d4760-4801-4ef0-83c8-f4ea8623c37c\" (UID: \"462d4760-4801-4ef0-83c8-f4ea8623c37c\") "
Nov 02 14:24:01.027661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.027644    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a97b65-d1cc-49c8-8816-133bbf7c8d96-kube-api-access-x9lhq\" (UniqueName: \"kubernetes.io/projected/96a97b65-d1cc-49c8-8816-133bbf7c8d96-kube-api-access-x9lhq\") pod \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\" (UID: \"96a97b65-d1cc-49c8-8816-133bbf7c8d96\") "
Nov 02 14:24:01.027737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.027721    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462d4760-4801-4ef0-83c8-f4ea8623c37c-kube-api-access-dvdwq\" (UniqueName: \"kubernetes.io/projected/462d4760-4801-4ef0-83c8-f4ea8623c37c-kube-api-access-dvdwq\") pod \"462d4760-4801-4ef0-83c8-f4ea8623c37c\" (UID: \"462d4760-4801-4ef0-83c8-f4ea8623c37c\") "
Nov 02 14:24:01.027827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.027749    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a97b65-d1cc-49c8-8816-133bbf7c8d96-plugins-config-volume" pod "96a97b65-d1cc-49c8-8816-133bbf7c8d96" (UID: "96a97b65-d1cc-49c8-8816-133bbf7c8d96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:24:01.028063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.028036    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462d4760-4801-4ef0-83c8-f4ea8623c37c-plugins-config-volume" pod "462d4760-4801-4ef0-83c8-f4ea8623c37c" (UID: "462d4760-4801-4ef0-83c8-f4ea8623c37c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:24:01.028443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.028426    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a97b65-d1cc-49c8-8816-133bbf7c8d96-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:01.028509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.028445    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462d4760-4801-4ef0-83c8-f4ea8623c37c-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:01.029928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.029906    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462d4760-4801-4ef0-83c8-f4ea8623c37c-kube-api-access-dvdwq" pod "462d4760-4801-4ef0-83c8-f4ea8623c37c" (UID: "462d4760-4801-4ef0-83c8-f4ea8623c37c"). InnerVolumeSpecName "kube-api-access-dvdwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:24:01.030004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.029942    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a97b65-d1cc-49c8-8816-133bbf7c8d96-kube-api-access-x9lhq" pod "96a97b65-d1cc-49c8-8816-133bbf7c8d96" (UID: "96a97b65-d1cc-49c8-8816-133bbf7c8d96"). InnerVolumeSpecName "kube-api-access-x9lhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:24:01.129540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.129464    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9lhq\" (UniqueName: \"kubernetes.io/projected/96a97b65-d1cc-49c8-8816-133bbf7c8d96-kube-api-access-x9lhq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:01.129540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.129489    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdwq\" (UniqueName: \"kubernetes.io/projected/462d4760-4801-4ef0-83c8-f4ea8623c37c-kube-api-access-dvdwq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:01.416362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.416331    5034 generic.go:356] "Generic (PLEG): container finished" podID="462d4760-4801-4ef0-83c8-f4ea8623c37c" containerID="735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc" exitCode=0
Nov 02 14:24:01.416535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.416411    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"
Nov 02 14:24:01.416535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.416417    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs" event={"ID":"462d4760-4801-4ef0-83c8-f4ea8623c37c","Type":"ContainerDied","Data":"735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc"}
Nov 02 14:24:01.416535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.416477    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs" event={"ID":"462d4760-4801-4ef0-83c8-f4ea8623c37c","Type":"ContainerDied","Data":"a7dd0f40faf78d059c3ee796d801e69a3b0e4a338ee6a034ed885f551b25bf62"}
Nov 02 14:24:01.416535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.416503    5034 scope.go:122] "RemoveContainer" containerID="735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc"
Nov 02 14:24:01.418127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.418084    5034 generic.go:356] "Generic (PLEG): container finished" podID="96a97b65-d1cc-49c8-8816-133bbf7c8d96" containerID="dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88" exitCode=0
Nov 02 14:24:01.418127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.418115    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz" event={"ID":"96a97b65-d1cc-49c8-8816-133bbf7c8d96","Type":"ContainerDied","Data":"dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88"}
Nov 02 14:24:01.418385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.418148    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"
Nov 02 14:24:01.418385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.418166    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz" event={"ID":"96a97b65-d1cc-49c8-8816-133bbf7c8d96","Type":"ContainerDied","Data":"6502f32ff48e1a05e1021e50f71a3fc4bef1929d1c95b479c626e3b02686f9ff"}
Nov 02 14:24:01.429441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.429422    5034 scope.go:122] "RemoveContainer" containerID="735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc"
Nov 02 14:24:01.429689 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:24:01.429671    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc\": container with ID starting with 735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc not found: ID does not exist" containerID="735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc"
Nov 02 14:24:01.429738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.429696    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc"} err="failed to get container status \"735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc\": rpc error: code = NotFound desc = could not find container \"735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc\": container with ID starting with 735581debcd47810f5624036e6c033efd98d1676a68e1b75923ec3d96c3cdadc not found: ID does not exist"
Nov 02 14:24:01.429738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.429712    5034 scope.go:122] "RemoveContainer" containerID="dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88"
Nov 02 14:24:01.440279 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.440264    5034 scope.go:122] "RemoveContainer" containerID="dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88"
Nov 02 14:24:01.440525 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:24:01.440508    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88\": container with ID starting with dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88 not found: ID does not exist" containerID="dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88"
Nov 02 14:24:01.440595 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.440531    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88"} err="failed to get container status \"dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88\": rpc error: code = NotFound desc = could not find container \"dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88\": container with ID starting with dccc4f0a850a8cb9348ae9861e2ae720c37deabbb3a4262b42fd6f7ef9473d88 not found: ID does not exist"
Nov 02 14:24:01.485901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.485877    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"]
Nov 02 14:24:01.499474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.499448    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-0c2554-6b67c855b7-zx9qs"]
Nov 02 14:24:01.526291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.526263    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"]
Nov 02 14:24:01.530408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:01.530389    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0c2554-774886bbd7-m79bz"]
Nov 02 14:24:03.400022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:03.399985    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462d4760-4801-4ef0-83c8-f4ea8623c37c" path="/var/lib/kubelet/pods/462d4760-4801-4ef0-83c8-f4ea8623c37c/volumes"
Nov 02 14:24:03.400664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:03.400644    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a97b65-d1cc-49c8-8816-133bbf7c8d96" path="/var/lib/kubelet/pods/96a97b65-d1cc-49c8-8816-133bbf7c8d96/volumes"
Nov 02 14:24:09.627554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.627517    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"]
Nov 02 14:24:09.628558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.628532    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy"
Nov 02 14:24:09.628558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.628558    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy"
Nov 02 14:24:09.628742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.628627    5034 cpu_manager.go:414] "containerMap: removing container" podUID="96a97b65-d1cc-49c8-8816-133bbf7c8d96" containerName="payload-pre-processing"
Nov 02 14:24:09.628742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.628637    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a97b65-d1cc-49c8-8816-133bbf7c8d96" containerName="payload-pre-processing"
Nov 02 14:24:09.628742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.628675    5034 cpu_manager.go:414] "containerMap: removing container" podUID="462d4760-4801-4ef0-83c8-f4ea8623c37c" containerName="payload-processing"
Nov 02 14:24:09.628742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.628681    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462d4760-4801-4ef0-83c8-f4ea8623c37c" containerName="payload-processing"
Nov 02 14:24:09.628978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.628903    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a97b65-d1cc-49c8-8816-133bbf7c8d96" containerName="payload-pre-processing"
Nov 02 14:24:09.628978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.628927    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="462d4760-4801-4ef0-83c8-f4ea8623c37c" containerName="payload-processing"
Nov 02 14:24:09.628978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.628944    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48deb00-e178-4318-a592-c53c7899bddf" containerName="istio-proxy"
Nov 02 14:24:09.634475 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.634440    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.650998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.650957    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bc34db\""
Nov 02 14:24:09.651190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.651022    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"routing-overlay\""
Nov 02 14:24:09.657302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.654580    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"]
Nov 02 14:24:09.674594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.673932    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.674594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.674068    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.674594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.674118    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqm7\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-kube-api-access-6zqm7\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.674594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.674194    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.684652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.684622    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"agc-cred-c56b8ba7682d\""
Nov 02 14:24:09.684847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.684672    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"payload-processing-external-model-e2e--2bdbc850-dockercfg-v8bp5\""
Nov 02 14:24:09.753803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.750968    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"kube-root-ca.crt\""
Nov 02 14:24:09.753803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.751097    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"openshift-service-ca.crt\""
Nov 02 14:24:09.778183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.778105    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.778499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.778448    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqm7\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-kube-api-access-6zqm7\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.778758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.778689    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.779199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.779135    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.782825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.782802    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.794537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.794485    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.824899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.824874    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqm7\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-kube-api-access-6zqm7\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.861203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.861155    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:09.949283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:09.949190    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:10.269576 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:24:10.269530    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cd5719_36ce_499e_922d_ed3a3eaf3b94.slice/crio-5b4903a522a552331c2e4c04527b8451bf0854c9e50b3d5393ae075a3faa0170 WatchSource:0}: Error finding container 5b4903a522a552331c2e4c04527b8451bf0854c9e50b3d5393ae075a3faa0170: Status 404 returned error can't find the container with id 5b4903a522a552331c2e4c04527b8451bf0854c9e50b3d5393ae075a3faa0170
Nov 02 14:24:10.271540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:10.271509    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"]
Nov 02 14:24:10.568578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:10.568478    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk" event={"ID":"b0cd5719-36ce-499e-922d-ed3a3eaf3b94","Type":"ContainerStarted","Data":"51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3"}
Nov 02 14:24:10.568578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:10.568546    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk" event={"ID":"b0cd5719-36ce-499e-922d-ed3a3eaf3b94","Type":"ContainerStarted","Data":"5b4903a522a552331c2e4c04527b8451bf0854c9e50b3d5393ae075a3faa0170"}
Nov 02 14:24:10.568873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:10.568734    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:10.617117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:10.617055    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk" podStartSLOduration=1.617037865 podStartE2EDuration="1.617037865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:10.616063558 +0000 UTC m=+2441.367367996" watchObservedRunningTime="2026-11-02 14:24:10.617037865 +0000 UTC m=+2441.368342303"
Nov 02 14:24:15.032938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:15.032902    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"]
Nov 02 14:24:15.048463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:15.048428    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8417d92-ht4rq"]
Nov 02 14:24:15.397141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:15.397067    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93f5d70-a180-4d84-a54d-7093e773b008" path="/var/lib/kubelet/pods/c93f5d70-a180-4d84-a54d-7093e773b008/volumes"
Nov 02 14:24:18.428666 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:24:18.428574    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:24:20.225395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:20.224238    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"]
Nov 02 14:24:20.225395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:20.224553    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy" containerID="cri-o://fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf" gracePeriod=30
Nov 02 14:24:21.580639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:21.580607    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:24:22.660206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.660173    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"]
Nov 02 14:24:22.667996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.667966    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.697144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.697114    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"]
Nov 02 14:24:22.704752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.704722    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ce0f55-openshift-default-dockercfg-ghzd9\""
Nov 02 14:24:22.783441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.783405    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7p9t\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-kube-api-access-z7p9t\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.783642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.783481    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.783642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.783621    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.783794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.783757    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.783869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.783848    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.783976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.783959    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.784037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.783984    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.784096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.784071    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.784160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.784129    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.885972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.885929    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.886247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.886224    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.886435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.886418    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.886622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.886603    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.886766 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.886749    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7p9t\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-kube-api-access-z7p9t\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.886984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.886965    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.887134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.887119    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.896798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.887326    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.897023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.897002    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.897653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.897630    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-data\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.906799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.899054    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istiod-ca-cert\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.907867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.907840    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.926466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.908245    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-credential-socket\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.926641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.908485    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-certs\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.955801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.944728    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-envoy\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.955801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.950145    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7p9t\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-kube-api-access-z7p9t\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.967339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.967311    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-token\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:22.968060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:22.968035    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-podinfo\") pod \"e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") " pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:23.023368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:23.023062    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:23.376294 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:24:23.376264    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cc2e99_f558_4e2c_be3a_ed110bebf2e9.slice/crio-6fc004d43f3e2d8b6921e76c297431c21a6a38745a2de7324b7187546060c8df WatchSource:0}: Error finding container 6fc004d43f3e2d8b6921e76c297431c21a6a38745a2de7324b7187546060c8df: Status 404 returned error can't find the container with id 6fc004d43f3e2d8b6921e76c297431c21a6a38745a2de7324b7187546060c8df
Nov 02 14:24:23.377637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:23.377608    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"]
Nov 02 14:24:23.379437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:23.379399    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:24:23.379524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:23.379483    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:24:23.379566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:23.379524    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:24:23.714134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:23.714039    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" event={"ID":"23cc2e99-f558-4e2c-be3a-ed110bebf2e9","Type":"ContainerStarted","Data":"011a8d5633394df0e612db47dd7fe4c68859667ca1e4c756135c79f580b17a6e"}
Nov 02 14:24:23.714134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:23.714108    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" event={"ID":"23cc2e99-f558-4e2c-be3a-ed110bebf2e9","Type":"ContainerStarted","Data":"6fc004d43f3e2d8b6921e76c297431c21a6a38745a2de7324b7187546060c8df"}
Nov 02 14:24:23.766243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:23.766177    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" podStartSLOduration=1.766157702 podStartE2EDuration="1.766157702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:23.762696958 +0000 UTC m=+2454.514001397" watchObservedRunningTime="2026-11-02 14:24:23.766157702 +0000 UTC m=+2454.517462142"
Nov 02 14:24:24.023905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:24.023871    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:24.030683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:24.030651    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:24.722687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:24.722645    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:24.723679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:24.723658    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:24:25.031973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.031943    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"]
Nov 02 14:24:25.036196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.036171    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb9aa177-dpdld"]
Nov 02 14:24:25.399354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.399324    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c" path="/var/lib/kubelet/pods/a7ed2bf9-9f85-4731-bfd4-c1f7503caa9c/volumes"
Nov 02 14:24:25.583664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.583637    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:24:25.738646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.738608    5034 generic.go:356] "Generic (PLEG): container finished" podID="c074b624-bef7-47b6-b3cc-e390df934572" containerID="fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf" exitCode=0
Nov 02 14:24:25.739168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.738672    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"
Nov 02 14:24:25.739168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.738711    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" event={"ID":"c074b624-bef7-47b6-b3cc-e390df934572","Type":"ContainerDied","Data":"fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf"}
Nov 02 14:24:25.739168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.738806    5034 scope.go:122] "RemoveContainer" containerID="fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf"
Nov 02 14:24:25.739168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.738961    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb" event={"ID":"c074b624-bef7-47b6-b3cc-e390df934572","Type":"ContainerDied","Data":"8b2deac215f35940c15262a066c29960ef9bc16f652e7bf26ce767d39f1ceea1"}
Nov 02 14:24:25.751927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.751907    5034 scope.go:122] "RemoveContainer" containerID="3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6"
Nov 02 14:24:25.762389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762260    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-kube-api-access-wwsvf\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-kube-api-access-wwsvf\") pod \"c074b624-bef7-47b6-b3cc-e390df934572\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") "
Nov 02 14:24:25.762389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762344    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-envoy\") pod \"c074b624-bef7-47b6-b3cc-e390df934572\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") "
Nov 02 14:24:25.762509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762425    5034 scope.go:122] "RemoveContainer" containerID="fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf"
Nov 02 14:24:25.762509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762440    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c074b624-bef7-47b6-b3cc-e390df934572-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c074b624-bef7-47b6-b3cc-e390df934572-istiod-ca-cert\") pod \"c074b624-bef7-47b6-b3cc-e390df934572\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") "
Nov 02 14:24:25.762509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762485    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-credential-socket\") pod \"c074b624-bef7-47b6-b3cc-e390df934572\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") "
Nov 02 14:24:25.762674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762576    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c074b624-bef7-47b6-b3cc-e390df934572-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c074b624-bef7-47b6-b3cc-e390df934572-istio-podinfo\") pod \"c074b624-bef7-47b6-b3cc-e390df934572\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") "
Nov 02 14:24:25.762674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762646    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-istio-token\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-istio-token\") pod \"c074b624-bef7-47b6-b3cc-e390df934572\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") "
Nov 02 14:24:25.762814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762693    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-certs\") pod \"c074b624-bef7-47b6-b3cc-e390df934572\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") "
Nov 02 14:24:25.762814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762744    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-credential-socket" pod "c074b624-bef7-47b6-b3cc-e390df934572" (UID: "c074b624-bef7-47b6-b3cc-e390df934572"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:24:25.762937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762854    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c074b624-bef7-47b6-b3cc-e390df934572-istiod-ca-cert" pod "c074b624-bef7-47b6-b3cc-e390df934572" (UID: "c074b624-bef7-47b6-b3cc-e390df934572"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:24:25.762937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762868    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-socket\") pod \"c074b624-bef7-47b6-b3cc-e390df934572\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") "
Nov 02 14:24:25.762937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.762920    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-data\") pod \"c074b624-bef7-47b6-b3cc-e390df934572\" (UID: \"c074b624-bef7-47b6-b3cc-e390df934572\") "
Nov 02 14:24:25.763413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763156    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-certs" pod "c074b624-bef7-47b6-b3cc-e390df934572" (UID: "c074b624-bef7-47b6-b3cc-e390df934572"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:24:25.763413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763345    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-socket" pod "c074b624-bef7-47b6-b3cc-e390df934572" (UID: "c074b624-bef7-47b6-b3cc-e390df934572"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:24:25.763413 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:24:25.763364    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf\": container with ID starting with fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf not found: ID does not exist" containerID="fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf"
Nov 02 14:24:25.763413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763382    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-data" pod "c074b624-bef7-47b6-b3cc-e390df934572" (UID: "c074b624-bef7-47b6-b3cc-e390df934572"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:24:25.763413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763400    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf"} err="failed to get container status \"fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf\": rpc error: code = NotFound desc = could not find container \"fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf\": container with ID starting with fbf59773e45640d830cced63a188f0003d0f115980b20da3fe3726fbffb9eebf not found: ID does not exist"
Nov 02 14:24:25.763702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763424    5034 scope.go:122] "RemoveContainer" containerID="3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6"
Nov 02 14:24:25.763824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763805    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c074b624-bef7-47b6-b3cc-e390df934572-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:25.763883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763834    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:25.763883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763849    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:25.763883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763864    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:25.763883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.763878    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:25.765209 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:24:25.765150    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6\": container with ID starting with 3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6 not found: ID does not exist" containerID="3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6"
Nov 02 14:24:25.765209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.765188    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6"} err="failed to get container status \"3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6\": rpc error: code = NotFound desc = could not find container \"3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6\": container with ID starting with 3eae032be8721be3dea072f51bba19515261fffba3e496cf4d4927088a91d2b6 not found: ID does not exist"
Nov 02 14:24:25.765542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.765494    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-envoy" pod "c074b624-bef7-47b6-b3cc-e390df934572" (UID: "c074b624-bef7-47b6-b3cc-e390df934572"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:24:25.765542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.765496    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c074b624-bef7-47b6-b3cc-e390df934572-istio-podinfo" pod "c074b624-bef7-47b6-b3cc-e390df934572" (UID: "c074b624-bef7-47b6-b3cc-e390df934572"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:24:25.765674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.765543    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-istio-token" pod "c074b624-bef7-47b6-b3cc-e390df934572" (UID: "c074b624-bef7-47b6-b3cc-e390df934572"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:24:25.765674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.765642    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-kube-api-access-wwsvf" pod "c074b624-bef7-47b6-b3cc-e390df934572" (UID: "c074b624-bef7-47b6-b3cc-e390df934572"). InnerVolumeSpecName "kube-api-access-wwsvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:24:25.867746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.866822    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c074b624-bef7-47b6-b3cc-e390df934572-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:25.867746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.866855    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:25.867746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.866870    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwsvf\" (UniqueName: \"kubernetes.io/projected/c074b624-bef7-47b6-b3cc-e390df934572-kube-api-access-wwsvf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:25.867746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:25.866883    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c074b624-bef7-47b6-b3cc-e390df934572-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:26.097043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:26.097010    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"]
Nov 02 14:24:26.112642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:26.112607    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-0c2554-openshift-default-6ff96747fc-xzqvb"]
Nov 02 14:24:27.397371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:27.397334    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c074b624-bef7-47b6-b3cc-e390df934572" path="/var/lib/kubelet/pods/c074b624-bef7-47b6-b3cc-e390df934572/volumes"
Nov 02 14:24:28.138016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.137976    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"]
Nov 02 14:24:28.138993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.138970    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy"
Nov 02 14:24:28.138993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.138993    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy"
Nov 02 14:24:28.139148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.139005    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy"
Nov 02 14:24:28.139148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.139012    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy"
Nov 02 14:24:28.139231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.139163    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy"
Nov 02 14:24:28.139231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.139179    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c074b624-bef7-47b6-b3cc-e390df934572" containerName="istio-proxy"
Nov 02 14:24:28.144809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.144770    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:28.163740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.163706    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"]
Nov 02 14:24:28.175537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.175498    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ce0f55\""
Nov 02 14:24:28.211008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.194463    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ce0f55-dockercfg-mz6xb\""
Nov 02 14:24:28.384802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.369706    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"]
Nov 02 14:24:28.384802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.376007    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:28.384802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.379946    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"]
Nov 02 14:24:28.531900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.531860    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwbgb\" (UniqueName: \"kubernetes.io/projected/30e257b7-094d-48da-a548-3ec863b5f148-kube-api-access-dwbgb\") pod \"payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr\" (UID: \"30e257b7-094d-48da-a548-3ec863b5f148\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:28.532343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.532089    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e257b7-094d-48da-a548-3ec863b5f148-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr\" (UID: \"30e257b7-094d-48da-a548-3ec863b5f148\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:28.633470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.633432    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwbgb\" (UniqueName: \"kubernetes.io/projected/30e257b7-094d-48da-a548-3ec863b5f148-kube-api-access-dwbgb\") pod \"payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr\" (UID: \"30e257b7-094d-48da-a548-3ec863b5f148\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:28.633666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.633508    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf\" (UID: \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:28.633666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.633543    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e257b7-094d-48da-a548-3ec863b5f148-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr\" (UID: \"30e257b7-094d-48da-a548-3ec863b5f148\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:28.633836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.633806    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvs75\" (UniqueName: \"kubernetes.io/projected/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-kube-api-access-dvs75\") pod \"payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf\" (UID: \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:28.634338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.634314    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e257b7-094d-48da-a548-3ec863b5f148-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr\" (UID: \"30e257b7-094d-48da-a548-3ec863b5f148\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:28.674082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.674029    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwbgb\" (UniqueName: \"kubernetes.io/projected/30e257b7-094d-48da-a548-3ec863b5f148-kube-api-access-dwbgb\") pod \"payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr\" (UID: \"30e257b7-094d-48da-a548-3ec863b5f148\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:28.735232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.735198    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf\" (UID: \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:28.735410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.735333    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvs75\" (UniqueName: \"kubernetes.io/projected/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-kube-api-access-dvs75\") pod \"payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf\" (UID: \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:28.735980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.735957    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf\" (UID: \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:28.760396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.760372    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvs75\" (UniqueName: \"kubernetes.io/projected/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-kube-api-access-dvs75\") pod \"payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf\" (UID: \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:28.765227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:28.765202    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:29.010906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.010878    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:29.078315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.077908    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"]
Nov 02 14:24:29.334377 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:24:29.334348    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3805ee4d_ff1c_47f8_bd49_00b7b21e4eac.slice/crio-09c6838a34dbddf26a07bf3946e55cb211c4735a6160e69843dbd6b63f3c31a9 WatchSource:0}: Error finding container 09c6838a34dbddf26a07bf3946e55cb211c4735a6160e69843dbd6b63f3c31a9: Status 404 returned error can't find the container with id 09c6838a34dbddf26a07bf3946e55cb211c4735a6160e69843dbd6b63f3c31a9
Nov 02 14:24:29.335938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.335911    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"]
Nov 02 14:24:29.822145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.822110    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf" event={"ID":"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac","Type":"ContainerStarted","Data":"1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698"}
Nov 02 14:24:29.822648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.822174    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf" event={"ID":"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac","Type":"ContainerStarted","Data":"09c6838a34dbddf26a07bf3946e55cb211c4735a6160e69843dbd6b63f3c31a9"}
Nov 02 14:24:29.822648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.822203    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:29.823995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.823963    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr" event={"ID":"30e257b7-094d-48da-a548-3ec863b5f148","Type":"ContainerStarted","Data":"295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39"}
Nov 02 14:24:29.824130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.824020    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr" event={"ID":"30e257b7-094d-48da-a548-3ec863b5f148","Type":"ContainerStarted","Data":"e4bc61ce425e679825cab773b37820549786884cc2928cb1b04cc3ba536f7bc8"}
Nov 02 14:24:29.824130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.824052    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:29.867430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.867386    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf" podStartSLOduration=1.867372493 podStartE2EDuration="1.867372493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:29.866388414 +0000 UTC m=+2460.617692851" watchObservedRunningTime="2026-11-02 14:24:29.867372493 +0000 UTC m=+2460.618676961"
Nov 02 14:24:29.910914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:29.910861    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr" podStartSLOduration=1.910846349 podStartE2EDuration="1.910846349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:29.909961871 +0000 UTC m=+2460.661266313" watchObservedRunningTime="2026-11-02 14:24:29.910846349 +0000 UTC m=+2460.662150786"
Nov 02 14:24:34.345074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.345040    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"]
Nov 02 14:24:34.350229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.350206    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.372861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.372836    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"]
Nov 02 14:24:34.379339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.379314    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-b1ab1ce91238\""
Nov 02 14:24:34.441641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.441608    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlcwh\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-kube-api-access-hlcwh\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.441799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.441663    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.441880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.441854    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.442100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.442077    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.543408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.543369    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlcwh\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-kube-api-access-hlcwh\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.543568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.543435    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.543568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.543502    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.543699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.543636    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.544095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.544066    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.544218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.544191    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.546241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.546212    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.568501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.568480    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlcwh\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-kube-api-access-hlcwh\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.665578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.665518    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:34.979293 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:24:34.979264    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181590bb_3370_4a82_9c04_3e1a3af9840c.slice/crio-22777ab676956a2286230c051b10ca6f89e7081d7197ed26e0d86d13bf067976 WatchSource:0}: Error finding container 22777ab676956a2286230c051b10ca6f89e7081d7197ed26e0d86d13bf067976: Status 404 returned error can't find the container with id 22777ab676956a2286230c051b10ca6f89e7081d7197ed26e0d86d13bf067976
Nov 02 14:24:34.983139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:34.983106    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"]
Nov 02 14:24:35.886204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:35.886163    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh" event={"ID":"181590bb-3370-4a82-9c04-3e1a3af9840c","Type":"ContainerStarted","Data":"0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431"}
Nov 02 14:24:35.886731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:35.886227    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh" event={"ID":"181590bb-3370-4a82-9c04-3e1a3af9840c","Type":"ContainerStarted","Data":"22777ab676956a2286230c051b10ca6f89e7081d7197ed26e0d86d13bf067976"}
Nov 02 14:24:35.886731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:35.886293    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:35.932637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:35.932594    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh" podStartSLOduration=1.9325796849999999 podStartE2EDuration="1.932579685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:24:35.931977974 +0000 UTC m=+2466.683282412" watchObservedRunningTime="2026-11-02 14:24:35.932579685 +0000 UTC m=+2466.683884125"
Nov 02 14:24:40.837858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:40.837752    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:24:40.838878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:40.838856    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:24:46.901569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:46.901540    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:24:46.978510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:46.978475    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"]
Nov 02 14:24:48.415483 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:24:48.415444    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:24:51.979088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:51.979048    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb" podUID="521cf372-2048-44b9-9fe3-67e3d9f42664" containerName="payload-processing" containerID="cri-o://07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a" gracePeriod=25
Nov 02 14:24:52.272963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.272938    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:24:52.307933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.307904    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-plugins-config-volume\") pod \"521cf372-2048-44b9-9fe3-67e3d9f42664\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") "
Nov 02 14:24:52.308058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.307959    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-routing-overlay-volume\") pod \"521cf372-2048-44b9-9fe3-67e3d9f42664\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") "
Nov 02 14:24:52.308130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.308076    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-kube-api-access-mthwp\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-kube-api-access-mthwp\") pod \"521cf372-2048-44b9-9fe3-67e3d9f42664\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") "
Nov 02 14:24:52.308292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.308255    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-provider-credentials\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-provider-credentials\") pod \"521cf372-2048-44b9-9fe3-67e3d9f42664\" (UID: \"521cf372-2048-44b9-9fe3-67e3d9f42664\") "
Nov 02 14:24:52.308414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.308375    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-routing-overlay-volume" pod "521cf372-2048-44b9-9fe3-67e3d9f42664" (UID: "521cf372-2048-44b9-9fe3-67e3d9f42664"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:24:52.308495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.308385    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-plugins-config-volume" pod "521cf372-2048-44b9-9fe3-67e3d9f42664" (UID: "521cf372-2048-44b9-9fe3-67e3d9f42664"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:24:52.308890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.308864    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:52.309002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.308892    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/521cf372-2048-44b9-9fe3-67e3d9f42664-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:52.310543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.310515    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-provider-credentials" pod "521cf372-2048-44b9-9fe3-67e3d9f42664" (UID: "521cf372-2048-44b9-9fe3-67e3d9f42664"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:24:52.310819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.310795    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-kube-api-access-mthwp" pod "521cf372-2048-44b9-9fe3-67e3d9f42664" (UID: "521cf372-2048-44b9-9fe3-67e3d9f42664"). InnerVolumeSpecName "kube-api-access-mthwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:24:52.410736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.410633    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:52.410736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:52.410665    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mthwp\" (UniqueName: \"kubernetes.io/projected/521cf372-2048-44b9-9fe3-67e3d9f42664-kube-api-access-mthwp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:24:53.051550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.051516    5034 generic.go:356] "Generic (PLEG): container finished" podID="521cf372-2048-44b9-9fe3-67e3d9f42664" containerID="07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a" exitCode=0
Nov 02 14:24:53.052030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.051566    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"
Nov 02 14:24:53.052030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.051593    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb" event={"ID":"521cf372-2048-44b9-9fe3-67e3d9f42664","Type":"ContainerDied","Data":"07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a"}
Nov 02 14:24:53.052030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.051644    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb" event={"ID":"521cf372-2048-44b9-9fe3-67e3d9f42664","Type":"ContainerDied","Data":"410a087a49b4aa835e2e37148aa8091f987b259b591a5cd22a978d97de204054"}
Nov 02 14:24:53.052030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.051678    5034 scope.go:122] "RemoveContainer" containerID="07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a"
Nov 02 14:24:53.063421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.063403    5034 scope.go:122] "RemoveContainer" containerID="07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a"
Nov 02 14:24:53.063694 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:24:53.063674    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a\": container with ID starting with 07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a not found: ID does not exist" containerID="07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a"
Nov 02 14:24:53.063756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.063702    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a"} err="failed to get container status \"07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a\": rpc error: code = NotFound desc = could not find container \"07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a\": container with ID starting with 07eddfb5ca7a2aac9264fff05cbc197a3720c7d24b34c38832cd3c9a1e0d661a not found: ID does not exist"
Nov 02 14:24:53.116424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.116385    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"]
Nov 02 14:24:53.125602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.125574    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-6bd5kf6bb"]
Nov 02 14:24:53.398936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:24:53.398857    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521cf372-2048-44b9-9fe3-67e3d9f42664" path="/var/lib/kubelet/pods/521cf372-2048-44b9-9fe3-67e3d9f42664/volumes"
Nov 02 14:25:01.087804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:01.087481    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"]
Nov 02 14:25:01.113802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:01.105356    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3f2936f-xnnbf"]
Nov 02 14:25:01.424213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:01.424136    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c34dd58-c602-4045-a31c-54e8bbec432f" path="/var/lib/kubelet/pods/9c34dd58-c602-4045-a31c-54e8bbec432f/volumes"
Nov 02 14:25:03.210269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.210236    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"]
Nov 02 14:25:03.211222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.211202    5034 cpu_manager.go:414] "containerMap: removing container" podUID="521cf372-2048-44b9-9fe3-67e3d9f42664" containerName="payload-processing"
Nov 02 14:25:03.211222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.211221    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521cf372-2048-44b9-9fe3-67e3d9f42664" containerName="payload-processing"
Nov 02 14:25:03.211384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.211375    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="521cf372-2048-44b9-9fe3-67e3d9f42664" containerName="payload-processing"
Nov 02 14:25:03.216596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.216578    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.237676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.237654    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"]
Nov 02 14:25:03.247049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.247019    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a02c35\""
Nov 02 14:25:03.247695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.247674    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"openshift-service-ca.crt\""
Nov 02 14:25:03.247847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.247743    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"routing-overlay\""
Nov 02 14:25:03.247847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.247833    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"kube-root-ca.crt\""
Nov 02 14:25:03.298070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.298039    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"agc-cred-addf94a36aad\""
Nov 02 14:25:03.298070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.298039    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"agc-cred-d10ecd76f087\""
Nov 02 14:25:03.298317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.298159    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-e2e--ac2d6d70-dockercfg-gjz62\""
Nov 02 14:25:03.356927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.356894    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.357086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.356939    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbw2v\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-kube-api-access-xbw2v\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.357086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.357063    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.357206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.357193    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.458540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.458503    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.458737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.458640    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.458737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.458727    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.458928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.458766    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbw2v\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-kube-api-access-xbw2v\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.459344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.459316    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.459446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.459372    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.461405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.461358    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.494501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.494471    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbw2v\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-kube-api-access-xbw2v\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.531898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.531871    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:03.862997 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:25:03.862962    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fbddffa_9066_4d71_ace7_39b307a80d37.slice/crio-20144f8f3a6518fdf75dcfba6ccead9b94d05ef746bab9c28e1ec97427c5139c WatchSource:0}: Error finding container 20144f8f3a6518fdf75dcfba6ccead9b94d05ef746bab9c28e1ec97427c5139c: Status 404 returned error can't find the container with id 20144f8f3a6518fdf75dcfba6ccead9b94d05ef746bab9c28e1ec97427c5139c
Nov 02 14:25:03.864841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:03.864817    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"]
Nov 02 14:25:04.213944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:04.213845    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b" event={"ID":"8fbddffa-9066-4d71-ace7-39b307a80d37","Type":"ContainerStarted","Data":"edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5"}
Nov 02 14:25:04.213944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:04.213918    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b" event={"ID":"8fbddffa-9066-4d71-ace7-39b307a80d37","Type":"ContainerStarted","Data":"20144f8f3a6518fdf75dcfba6ccead9b94d05ef746bab9c28e1ec97427c5139c"}
Nov 02 14:25:04.214424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:04.213951    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:04.264599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:04.264546    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b" podStartSLOduration=1.264532236 podStartE2EDuration="1.264532236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:04.261074197 +0000 UTC m=+2495.012378636" watchObservedRunningTime="2026-11-02 14:25:04.264532236 +0000 UTC m=+2495.015836673"
Nov 02 14:25:07.960919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:07.960884    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:25:07.961420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:07.961018    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:25:09.258995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:09.258961    5034 generic.go:356] "Generic (PLEG): container finished" podID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerID="2e4f925401a8fa0f3a41f51c03d79d7539dde864271740a4e48333eec76e6773" exitCode=0
Nov 02 14:25:09.259432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:09.259044    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" event={"ID":"edbee88c-756f-40fe-a43f-3da83ea117a7","Type":"ContainerDied","Data":"2e4f925401a8fa0f3a41f51c03d79d7539dde864271740a4e48333eec76e6773"}
Nov 02 14:25:09.259484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:09.259459    5034 scope.go:122] "RemoveContainer" containerID="2e4f925401a8fa0f3a41f51c03d79d7539dde864271740a4e48333eec76e6773"
Nov 02 14:25:09.261265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:09.261239    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:25:09.261342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:09.261307    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:25:09.261380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:09.261357    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:25:10.292520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:10.292477    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" event={"ID":"edbee88c-756f-40fe-a43f-3da83ea117a7","Type":"ContainerStarted","Data":"1ee3c666127e53a74aa9cf984536a6559e31ed1c571e9bbee2dc452f787283dd"}
Nov 02 14:25:10.405326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:10.397291    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:25:10.405326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:10.397395    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Nov 02 14:25:10.405326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:10.397434    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Nov 02 14:25:11.401802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:11.401259    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Nov 02 14:25:11.401802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:11.401345    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Nov 02 14:25:12.032651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:12.032616    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"]
Nov 02 14:25:12.046041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:12.046012    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f151b97-hmfwd"]
Nov 02 14:25:12.110235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:12.110205    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"]
Nov 02 14:25:12.110657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:12.110626    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy" containerID="cri-o://7adad05e0c6fb77909636bc4d73bede1c7c87e37f50e80dca2792b4cdad46079" gracePeriod=30
Nov 02 14:25:12.384228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:12.384136    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Nov 02 14:25:12.384486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:12.384457    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Nov 02 14:25:13.385179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:13.385144    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Nov 02 14:25:13.385768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:13.385734    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Nov 02 14:25:13.414552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:13.414523    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4828403e-b64b-42f8-ad57-7c9db90c5c26" path="/var/lib/kubelet/pods/4828403e-b64b-42f8-ad57-7c9db90c5c26/volumes"
Nov 02 14:25:14.391801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:14.391186    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Nov 02 14:25:14.391801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:14.391277    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Nov 02 14:25:15.227617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:15.227586    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:25:15.383821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:15.383765    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Nov 02 14:25:15.384000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:15.383850    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Nov 02 14:25:15.881432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:15.881387    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"]
Nov 02 14:25:16.388740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:16.388713    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:25:16.389286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:16.389257    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:25:16.390028 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:16.390005    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:25:17.372286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.372251    5034 generic.go:356] "Generic (PLEG): container finished" podID="444082ae-6bfd-4126-8ae1-f30e98929081" containerID="7adad05e0c6fb77909636bc4d73bede1c7c87e37f50e80dca2792b4cdad46079" exitCode=0
Nov 02 14:25:17.372768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.372678    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" event={"ID":"444082ae-6bfd-4126-8ae1-f30e98929081","Type":"ContainerDied","Data":"7adad05e0c6fb77909636bc4d73bede1c7c87e37f50e80dca2792b4cdad46079"}
Nov 02 14:25:17.372768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.372749    5034 scope.go:122] "RemoveContainer" containerID="c0e9bc5b929fd4d895707b2f47678c15e1cbb737b37e5cddce3d389d65cf59ac"
Nov 02 14:25:17.422463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.422438    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:25:17.574948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.574919    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-kube-api-access-d5tf7\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-kube-api-access-d5tf7\") pod \"444082ae-6bfd-4126-8ae1-f30e98929081\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") "
Nov 02 14:25:17.575171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.574971    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-istio-token\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-istio-token\") pod \"444082ae-6bfd-4126-8ae1-f30e98929081\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") "
Nov 02 14:25:17.575171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.575022    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/444082ae-6bfd-4126-8ae1-f30e98929081-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444082ae-6bfd-4126-8ae1-f30e98929081-istio-podinfo\") pod \"444082ae-6bfd-4126-8ae1-f30e98929081\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") "
Nov 02 14:25:17.575171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.575119    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-certs\") pod \"444082ae-6bfd-4126-8ae1-f30e98929081\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") "
Nov 02 14:25:17.575365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.575201    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-socket\") pod \"444082ae-6bfd-4126-8ae1-f30e98929081\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") "
Nov 02 14:25:17.575365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.575259    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-envoy\") pod \"444082ae-6bfd-4126-8ae1-f30e98929081\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") "
Nov 02 14:25:17.575480 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.575402    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/444082ae-6bfd-4126-8ae1-f30e98929081-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444082ae-6bfd-4126-8ae1-f30e98929081-istiod-ca-cert\") pod \"444082ae-6bfd-4126-8ae1-f30e98929081\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") "
Nov 02 14:25:17.575480 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.575439    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-certs" pod "444082ae-6bfd-4126-8ae1-f30e98929081" (UID: "444082ae-6bfd-4126-8ae1-f30e98929081"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:17.575480 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.575467    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-data\") pod \"444082ae-6bfd-4126-8ae1-f30e98929081\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") "
Nov 02 14:25:17.575649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.575553    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-credential-socket\") pod \"444082ae-6bfd-4126-8ae1-f30e98929081\" (UID: \"444082ae-6bfd-4126-8ae1-f30e98929081\") "
Nov 02 14:25:17.576124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.576093    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-data" pod "444082ae-6bfd-4126-8ae1-f30e98929081" (UID: "444082ae-6bfd-4126-8ae1-f30e98929081"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:17.576704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.576254    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-socket" pod "444082ae-6bfd-4126-8ae1-f30e98929081" (UID: "444082ae-6bfd-4126-8ae1-f30e98929081"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:17.576704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.576506    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-credential-socket" pod "444082ae-6bfd-4126-8ae1-f30e98929081" (UID: "444082ae-6bfd-4126-8ae1-f30e98929081"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:17.576704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.576569    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/444082ae-6bfd-4126-8ae1-f30e98929081-istiod-ca-cert" pod "444082ae-6bfd-4126-8ae1-f30e98929081" (UID: "444082ae-6bfd-4126-8ae1-f30e98929081"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:25:17.576704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.576644    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:17.576704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.576664    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:17.576704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.576679    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:17.585121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.585053    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/444082ae-6bfd-4126-8ae1-f30e98929081-istio-podinfo" pod "444082ae-6bfd-4126-8ae1-f30e98929081" (UID: "444082ae-6bfd-4126-8ae1-f30e98929081"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:25:17.587170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.587139    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-kube-api-access-d5tf7" pod "444082ae-6bfd-4126-8ae1-f30e98929081" (UID: "444082ae-6bfd-4126-8ae1-f30e98929081"). InnerVolumeSpecName "kube-api-access-d5tf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:17.588231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.588202    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-istio-token" pod "444082ae-6bfd-4126-8ae1-f30e98929081" (UID: "444082ae-6bfd-4126-8ae1-f30e98929081"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:17.590017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.589373    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-envoy" pod "444082ae-6bfd-4126-8ae1-f30e98929081" (UID: "444082ae-6bfd-4126-8ae1-f30e98929081"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:25:17.678812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.678736    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/444082ae-6bfd-4126-8ae1-f30e98929081-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:17.678812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.678768    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:17.678812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.678796    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5tf7\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-kube-api-access-d5tf7\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:17.678812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.678811    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/444082ae-6bfd-4126-8ae1-f30e98929081-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:17.679009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.678825    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/444082ae-6bfd-4126-8ae1-f30e98929081-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:17.679009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:17.678837    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/444082ae-6bfd-4126-8ae1-f30e98929081-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:18.384704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:18.384673    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"
Nov 02 14:25:18.385142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:18.384708    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks" event={"ID":"444082ae-6bfd-4126-8ae1-f30e98929081","Type":"ContainerDied","Data":"c81e560e3a9c6dec985d583e4678409c1143fe644b959101e36afaaa1e304890"}
Nov 02 14:25:18.385142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:18.384788    5034 scope.go:122] "RemoveContainer" containerID="7adad05e0c6fb77909636bc4d73bede1c7c87e37f50e80dca2792b4cdad46079"
Nov 02 14:25:18.422100 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:25:18.422071    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:25:18.458110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:18.458082    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"]
Nov 02 14:25:18.472593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:18.472560    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-vw9ks"]
Nov 02 14:25:19.034731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:19.034692    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"]
Nov 02 14:25:19.046429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:19.046397    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d237115e-k8twg"]
Nov 02 14:25:19.402993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:19.402921    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" path="/var/lib/kubelet/pods/444082ae-6bfd-4126-8ae1-f30e98929081/volumes"
Nov 02 14:25:19.403605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:19.403577    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445dc6a8-5650-472e-8028-d9824aa09880" path="/var/lib/kubelet/pods/445dc6a8-5650-472e-8028-d9824aa09880/volumes"
Nov 02 14:25:20.889801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:20.888913    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh" podUID="181590bb-3370-4a82-9c04-3e1a3af9840c" containerName="payload-processing" containerID="cri-o://0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431" gracePeriod=25
Nov 02 14:25:21.211206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.211173    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:25:21.230364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.230328    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-routing-overlay-volume\") pod \"181590bb-3370-4a82-9c04-3e1a3af9840c\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") "
Nov 02 14:25:21.230522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.230380    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-provider-credentials\") pod \"181590bb-3370-4a82-9c04-3e1a3af9840c\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") "
Nov 02 14:25:21.230522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.230485    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-kube-api-access-hlcwh\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-kube-api-access-hlcwh\") pod \"181590bb-3370-4a82-9c04-3e1a3af9840c\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") "
Nov 02 14:25:21.230648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.230582    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-plugins-config-volume\") pod \"181590bb-3370-4a82-9c04-3e1a3af9840c\" (UID: \"181590bb-3370-4a82-9c04-3e1a3af9840c\") "
Nov 02 14:25:21.230921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.230889    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-routing-overlay-volume" pod "181590bb-3370-4a82-9c04-3e1a3af9840c" (UID: "181590bb-3370-4a82-9c04-3e1a3af9840c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:25:21.231065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.231041    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-plugins-config-volume" pod "181590bb-3370-4a82-9c04-3e1a3af9840c" (UID: "181590bb-3370-4a82-9c04-3e1a3af9840c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:25:21.231377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.231357    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:21.231483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.231382    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/181590bb-3370-4a82-9c04-3e1a3af9840c-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:21.232717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.232687    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-provider-credentials" pod "181590bb-3370-4a82-9c04-3e1a3af9840c" (UID: "181590bb-3370-4a82-9c04-3e1a3af9840c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:21.232835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.232767    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-kube-api-access-hlcwh" pod "181590bb-3370-4a82-9c04-3e1a3af9840c" (UID: "181590bb-3370-4a82-9c04-3e1a3af9840c"). InnerVolumeSpecName "kube-api-access-hlcwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:21.332485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.332454    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlcwh\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-kube-api-access-hlcwh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:21.332485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.332483    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/181590bb-3370-4a82-9c04-3e1a3af9840c-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:21.416724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.416686    5034 generic.go:356] "Generic (PLEG): container finished" podID="181590bb-3370-4a82-9c04-3e1a3af9840c" containerID="0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431" exitCode=0
Nov 02 14:25:21.416904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.416736    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"
Nov 02 14:25:21.416904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.416767    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh" event={"ID":"181590bb-3370-4a82-9c04-3e1a3af9840c","Type":"ContainerDied","Data":"0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431"}
Nov 02 14:25:21.416904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.416836    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh" event={"ID":"181590bb-3370-4a82-9c04-3e1a3af9840c","Type":"ContainerDied","Data":"22777ab676956a2286230c051b10ca6f89e7081d7197ed26e0d86d13bf067976"}
Nov 02 14:25:21.416904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.416887    5034 scope.go:122] "RemoveContainer" containerID="0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431"
Nov 02 14:25:21.429711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.429683    5034 scope.go:122] "RemoveContainer" containerID="0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431"
Nov 02 14:25:21.430118 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:25:21.430080    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431\": container with ID starting with 0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431 not found: ID does not exist" containerID="0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431"
Nov 02 14:25:21.430232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.430119    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431"} err="failed to get container status \"0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431\": rpc error: code = NotFound desc = could not find container \"0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431\": container with ID starting with 0f2cea49889753193d87f790fc1cee559be0545b3b532472ea10234e0b328431 not found: ID does not exist"
Nov 02 14:25:21.477767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.477726    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"]
Nov 02 14:25:21.526800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:21.526723    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bff7rkkh"]
Nov 02 14:25:23.399153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:23.399114    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181590bb-3370-4a82-9c04-3e1a3af9840c" path="/var/lib/kubelet/pods/181590bb-3370-4a82-9c04-3e1a3af9840c/volumes"
Nov 02 14:25:25.880023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.879972    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"]
Nov 02 14:25:25.880948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.880928    5034 cpu_manager.go:414] "containerMap: removing container" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy"
Nov 02 14:25:25.880948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.880948    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy"
Nov 02 14:25:25.881032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.880995    5034 cpu_manager.go:414] "containerMap: removing container" podUID="181590bb-3370-4a82-9c04-3e1a3af9840c" containerName="payload-processing"
Nov 02 14:25:25.881032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.881001    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181590bb-3370-4a82-9c04-3e1a3af9840c" containerName="payload-processing"
Nov 02 14:25:25.881148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.881138    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="181590bb-3370-4a82-9c04-3e1a3af9840c" containerName="payload-processing"
Nov 02 14:25:25.881186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.881157    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy"
Nov 02 14:25:25.881186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.881167    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy"
Nov 02 14:25:25.885340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.885321    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:25.904141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.904118    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"]
Nov 02 14:25:25.913127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:25.913104    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"agc-cred-4c908c3e55be\""
Nov 02 14:25:26.019734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.019702    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.019955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.019929    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.020116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.019977    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.020240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.020213    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm64w\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-kube-api-access-tm64w\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.121898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.121870    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm64w\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-kube-api-access-tm64w\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.122041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.121959    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.122096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.122056    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.122096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.122077    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.122748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.122728    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.122910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.122892    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.124617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.124594    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.147122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.147062    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm64w\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-kube-api-access-tm64w\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.200303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.200275    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:26.505708 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:25:26.505669    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c60ad12_034c_4362_bba6_d3e2ff33caf8.slice/crio-c968a2bec73950d3d75706dd3700ac823b720c201d909164b8fff04f40ebe52f WatchSource:0}: Error finding container c968a2bec73950d3d75706dd3700ac823b720c201d909164b8fff04f40ebe52f: Status 404 returned error can't find the container with id c968a2bec73950d3d75706dd3700ac823b720c201d909164b8fff04f40ebe52f
Nov 02 14:25:26.510161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:26.510136    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"]
Nov 02 14:25:27.469073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:27.469038    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb" event={"ID":"8c60ad12-034c-4362-bba6-d3e2ff33caf8","Type":"ContainerStarted","Data":"835c9f4026aaf264fa001d5b87409e45dd127abf62075e14c2b46748ad8aa1a3"}
Nov 02 14:25:27.469507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:27.469093    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb" event={"ID":"8c60ad12-034c-4362-bba6-d3e2ff33caf8","Type":"ContainerStarted","Data":"c968a2bec73950d3d75706dd3700ac823b720c201d909164b8fff04f40ebe52f"}
Nov 02 14:25:27.469507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:27.469154    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:27.517289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:27.517230    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb" podStartSLOduration=2.517212408 podStartE2EDuration="2.517212408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:27.512855002 +0000 UTC m=+2518.264159442" watchObservedRunningTime="2026-11-02 14:25:27.517212408 +0000 UTC m=+2518.268516847"
Nov 02 14:25:30.033327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:30.033294    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"]
Nov 02 14:25:30.050061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:30.050025    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d4da17-j2qdd"]
Nov 02 14:25:31.399864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:31.399826    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ccc56c-b977-4711-8940-c7101b802009" path="/var/lib/kubelet/pods/03ccc56c-b977-4711-8940-c7101b802009/volumes"
Nov 02 14:25:36.760082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:36.760049    5034 scope.go:122] "RemoveContainer" containerID="87056c376d77fb7a003b98b6da2ecf6d90bf463ad7477912ca9d8e973b424139"
Nov 02 14:25:36.772295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:36.772260    5034 scope.go:122] "RemoveContainer" containerID="20913736827a4d19a125c7d5d85ae5821199e3b8c7e75acb028e1d52cbb1fc4c"
Nov 02 14:25:38.483162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:38.483133    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:25:38.568440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:38.568408    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"]
Nov 02 14:25:43.575474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:43.575429    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk" podUID="b0cd5719-36ce-499e-922d-ed3a3eaf3b94" containerName="payload-processing" containerID="cri-o://51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3" gracePeriod=25
Nov 02 14:25:43.881276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:43.881244    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:25:44.016666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.016634    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-routing-overlay-volume\") pod \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") "
Nov 02 14:25:44.016854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.016713    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-plugins-config-volume\") pod \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") "
Nov 02 14:25:44.016854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.016843    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-kube-api-access-6zqm7\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-kube-api-access-6zqm7\") pod \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") "
Nov 02 14:25:44.016965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.016927    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-provider-credentials\") pod \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\" (UID: \"b0cd5719-36ce-499e-922d-ed3a3eaf3b94\") "
Nov 02 14:25:44.017082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.017059    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-routing-overlay-volume" pod "b0cd5719-36ce-499e-922d-ed3a3eaf3b94" (UID: "b0cd5719-36ce-499e-922d-ed3a3eaf3b94"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:25:44.017195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.017164    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-plugins-config-volume" pod "b0cd5719-36ce-499e-922d-ed3a3eaf3b94" (UID: "b0cd5719-36ce-499e-922d-ed3a3eaf3b94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:25:44.017679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.017655    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:44.017679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.017680    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:44.019293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.019269    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-provider-credentials" pod "b0cd5719-36ce-499e-922d-ed3a3eaf3b94" (UID: "b0cd5719-36ce-499e-922d-ed3a3eaf3b94"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:44.019376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.019308    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-kube-api-access-6zqm7" pod "b0cd5719-36ce-499e-922d-ed3a3eaf3b94" (UID: "b0cd5719-36ce-499e-922d-ed3a3eaf3b94"). InnerVolumeSpecName "kube-api-access-6zqm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:25:44.119049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.118969    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zqm7\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-kube-api-access-6zqm7\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:44.119049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.118995    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cd5719-36ce-499e-922d-ed3a3eaf3b94-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:25:44.627358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.627325    5034 generic.go:356] "Generic (PLEG): container finished" podID="b0cd5719-36ce-499e-922d-ed3a3eaf3b94" containerID="51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3" exitCode=0
Nov 02 14:25:44.627871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.627374    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"
Nov 02 14:25:44.627871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.627409    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk" event={"ID":"b0cd5719-36ce-499e-922d-ed3a3eaf3b94","Type":"ContainerDied","Data":"51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3"}
Nov 02 14:25:44.627871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.627472    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk" event={"ID":"b0cd5719-36ce-499e-922d-ed3a3eaf3b94","Type":"ContainerDied","Data":"5b4903a522a552331c2e4c04527b8451bf0854c9e50b3d5393ae075a3faa0170"}
Nov 02 14:25:44.627871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.627506    5034 scope.go:122] "RemoveContainer" containerID="51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3"
Nov 02 14:25:44.641810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.641789    5034 scope.go:122] "RemoveContainer" containerID="51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3"
Nov 02 14:25:44.642275 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:25:44.642244    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3\": container with ID starting with 51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3 not found: ID does not exist" containerID="51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3"
Nov 02 14:25:44.642395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.642285    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3"} err="failed to get container status \"51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3\": rpc error: code = NotFound desc = could not find container \"51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3\": container with ID starting with 51d2fe617118ff747c1d7844213f7c8a20b2d6a0c70dbf1f6cb560fb7d9e71e3 not found: ID does not exist"
Nov 02 14:25:44.686040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.686016    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"]
Nov 02 14:25:44.699534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:44.699514    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-7987tksbk"]
Nov 02 14:25:45.414479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:45.414443    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cd5719-36ce-499e-922d-ed3a3eaf3b94" path="/var/lib/kubelet/pods/b0cd5719-36ce-499e-922d-ed3a3eaf3b94/volumes"
Nov 02 14:25:48.416326 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:25:48.416286    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:25:49.031991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:49.031958    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"]
Nov 02 14:25:49.035856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:49.035828    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dece2c2-t26m9"]
Nov 02 14:25:49.414411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:49.414327    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f108e3a-366e-43a8-a0be-72ad201fde3d" path="/var/lib/kubelet/pods/0f108e3a-366e-43a8-a0be-72ad201fde3d/volumes"
Nov 02 14:25:54.824113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.824076    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"]
Nov 02 14:25:54.827737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.827708    5034 cpu_manager.go:414] "containerMap: removing container" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy"
Nov 02 14:25:54.827737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.827736    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="444082ae-6bfd-4126-8ae1-f30e98929081" containerName="istio-proxy"
Nov 02 14:25:54.827970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.827805    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b0cd5719-36ce-499e-922d-ed3a3eaf3b94" containerName="payload-processing"
Nov 02 14:25:54.827970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.827814    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cd5719-36ce-499e-922d-ed3a3eaf3b94" containerName="payload-processing"
Nov 02 14:25:54.828093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.828034    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cd5719-36ce-499e-922d-ed3a3eaf3b94" containerName="payload-processing"
Nov 02 14:25:54.834527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.834501    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"]
Nov 02 14:25:54.834725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.834613    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.851985 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.851960    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-66bc6d-openshift-default-dockercfg-phxcl\""
Nov 02 14:25:54.876161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.876132    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-credential-socket\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.876381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.876351    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-podinfo\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.876472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.876403    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-envoy\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.876536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.876515    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-data\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.876601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.876574    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-socket\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.876662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.876611    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-token\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.876662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.876642    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istiod-ca-cert\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.876927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.876898    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-certs\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.877035 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.877015    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf48q\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-kube-api-access-gf48q\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.978323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.978286    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-credential-socket\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.978492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.978401    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-podinfo\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.978492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.978433    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-envoy\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.978492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.978475    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-data\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.978691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.978521    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-socket\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.978691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.978554    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-token\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.978691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.978586    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istiod-ca-cert\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.978880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.978706    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-certs\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.978880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.978860    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf48q\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-kube-api-access-gf48q\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.979246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.979217    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-certs\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.979454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.979428    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istiod-ca-cert\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.979508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.979479    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-data\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.979556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.979527    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-credential-socket\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.979770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.979753    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-socket\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.980988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.980963    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-envoy\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:54.981361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:54.981339    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-podinfo\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:55.038732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.035641    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf48q\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-kube-api-access-gf48q\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:55.038732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.036104    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-token\") pod \"e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") " pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:55.158038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.157684    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:55.527627 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:25:55.527599    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f5b619_f69a_45ff_904a_d8507d2c2ea6.slice/crio-d15c53a7885cb840dbf2200f57242435e802592f401c8fc70e49f86560ee37d7 WatchSource:0}: Error finding container d15c53a7885cb840dbf2200f57242435e802592f401c8fc70e49f86560ee37d7: Status 404 returned error can't find the container with id d15c53a7885cb840dbf2200f57242435e802592f401c8fc70e49f86560ee37d7
Nov 02 14:25:55.529501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.529478    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"]
Nov 02 14:25:55.530336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.530305    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:25:55.530403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.530369    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:25:55.530403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.530399    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:25:55.734632 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.734597    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" event={"ID":"55f5b619-f69a-45ff-904a-d8507d2c2ea6","Type":"ContainerStarted","Data":"d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589"}
Nov 02 14:25:55.734842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.734658    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" event={"ID":"55f5b619-f69a-45ff-904a-d8507d2c2ea6","Type":"ContainerStarted","Data":"d15c53a7885cb840dbf2200f57242435e802592f401c8fc70e49f86560ee37d7"}
Nov 02 14:25:55.788116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:55.787992    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" podStartSLOduration=1.787973796 podStartE2EDuration="1.787973796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:25:55.787128304 +0000 UTC m=+2546.538432755" watchObservedRunningTime="2026-11-02 14:25:55.787973796 +0000 UTC m=+2546.539278235"
Nov 02 14:25:56.160001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:56.159909    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:56.161589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:56.161553    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 14:25:56.161756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:56.161608    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 14:25:57.164718 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:57.164689    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:57.853801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:57.850684    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:25:57.870386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:25:57.870342    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:26:00.017824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:00.017761    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"]
Nov 02 14:26:00.018317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:00.018030    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy" containerID="cri-o://4c9813146911ff714bfffb146239169043c41271742a8ada3e607337d544b99a" gracePeriod=30
Nov 02 14:26:00.125278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:00.125045    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"]
Nov 02 14:26:00.127483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:00.127247    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa3ec515-fwhm8"]
Nov 02 14:26:01.121015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.120980    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"]
Nov 02 14:26:01.127343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.127320    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.143850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.143824    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"]
Nov 02 14:26:01.152754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.152738    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"routing-overlay\""
Nov 02 14:26:01.153016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.152970    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2d13f4\""
Nov 02 14:26:01.153133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.153005    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"openshift-service-ca.crt\""
Nov 02 14:26:01.153133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.153117    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"kube-root-ca.crt\""
Nov 02 14:26:01.193805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.189580    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"agc-cred-3fbd4dc2e303\""
Nov 02 14:26:01.193805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.189719    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"payload-processing-external-model-e2e--1d26709d-dockercfg-v6xgj\""
Nov 02 14:26:01.228627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.228597    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.228799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.228695    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.228927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.228907    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.229083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.229058    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9p46\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-kube-api-access-m9p46\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.355801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.338976    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9p46\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-kube-api-access-m9p46\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.355801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.339178    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.355801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.339237    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.355801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.339387    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.367822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.358281    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.367822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.359245    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.383846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.381436    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.428797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.425629    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9p46\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-kube-api-access-m9p46\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.449119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.448401    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:01.474977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:01.472694    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f9c7dc6-e36f-435e-b95c-3beaa43781fb" path="/var/lib/kubelet/pods/9f9c7dc6-e36f-435e-b95c-3beaa43781fb/volumes"
Nov 02 14:26:02.347535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:02.347494    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"]
Nov 02 14:26:03.335107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.335063    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" event={"ID":"4b9252e8-f794-42a1-a041-dbbb5b70f40f","Type":"ContainerStarted","Data":"48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27"}
Nov 02 14:26:03.335406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.335379    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" event={"ID":"4b9252e8-f794-42a1-a041-dbbb5b70f40f","Type":"ContainerStarted","Data":"edea2a50318ca8b649529c939e9bf79389264f7688b557c552e299d24a09a1a3"}
Nov 02 14:26:03.336655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.336631    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:03.392803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.365530    5034 patch_prober.go:29] interesting pod/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:26:03.392803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.365594    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:26:03.416458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.416407    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" podStartSLOduration=2.416393642 podStartE2EDuration="2.416393642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:03.409451818 +0000 UTC m=+2554.160756270" watchObservedRunningTime="2026-11-02 14:26:03.416393642 +0000 UTC m=+2554.167698080"
Nov 02 14:26:03.822207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.822169    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"]
Nov 02 14:26:03.827230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.827210    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:03.847047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.847015    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"]
Nov 02 14:26:03.847328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:03.847301    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"agc-cred-75c10740976d\""
Nov 02 14:26:04.018211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.018177    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.018403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.018230    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vsk\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-kube-api-access-f2vsk\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.018403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.018368    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.018513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.018488    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.119643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.119545    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.119643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.119592    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vsk\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-kube-api-access-f2vsk\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.119956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.119659    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.119956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.119726    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.120500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.120467    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.121063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.121017    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.123963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.123940    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.144834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.144809    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vsk\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-kube-api-access-f2vsk\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.453853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:04.453007    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:04.579384 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:04.579235    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d13f4/agc-cred-3fbd4dc2e303: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:04.579605 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:04.579590    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:04.579748 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:04.579736    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials podName:4b9252e8-f794-42a1-a041-dbbb5b70f40f nodeName:}" failed. No retries permitted until 2026-11-02 14:26:05.079713555 +0000 UTC m=+2555.831017992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f") : secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:05.179097 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:05.155429    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d13f4/agc-cred-3fbd4dc2e303: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:05.179097 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:05.155457    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:05.179097 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:05.155516    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials podName:4b9252e8-f794-42a1-a041-dbbb5b70f40f nodeName:}" failed. No retries permitted until 2026-11-02 14:26:06.155496781 +0000 UTC m=+2556.906801216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f") : secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:05.261806 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:05.259942    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d97ee28_1ccf_4ce4_8c26_5c50a0ccf4ec.slice/crio-4c9813146911ff714bfffb146239169043c41271742a8ada3e607337d544b99a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d97ee28_1ccf_4ce4_8c26_5c50a0ccf4ec.slice/crio-conmon-4c9813146911ff714bfffb146239169043c41271742a8ada3e607337d544b99a.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 14:26:05.374214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.374174    5034 generic.go:356] "Generic (PLEG): container finished" podID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerID="4c9813146911ff714bfffb146239169043c41271742a8ada3e607337d544b99a" exitCode=0
Nov 02 14:26:05.374899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.374868    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" event={"ID":"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec","Type":"ContainerDied","Data":"4c9813146911ff714bfffb146239169043c41271742a8ada3e607337d544b99a"}
Nov 02 14:26:05.481655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.481601    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"]
Nov 02 14:26:05.482832 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:05.482801    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8007ba35_5dca_48bc_8401_62896c8af6e8.slice/crio-79acc9ff5eb3053f22f542fef05f704f983baf49f44eb7bb8b1cce14bd9f7197 WatchSource:0}: Error finding container 79acc9ff5eb3053f22f542fef05f704f983baf49f44eb7bb8b1cce14bd9f7197: Status 404 returned error can't find the container with id 79acc9ff5eb3053f22f542fef05f704f983baf49f44eb7bb8b1cce14bd9f7197
Nov 02 14:26:05.642315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.642287    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 14:26:05.664663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.664633    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-data\") pod \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") "
Nov 02 14:26:05.664809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.664706    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-podinfo\") pod \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") "
Nov 02 14:26:05.664864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.664825    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-data" pod "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" (UID: "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:26:05.664864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.664829    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-token\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-token\") pod \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") "
Nov 02 14:26:05.664959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.664890    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istiod-ca-cert\") pod \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") "
Nov 02 14:26:05.664959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.664918    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-socket\") pod \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") "
Nov 02 14:26:05.665062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.665001    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-credential-socket\") pod \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") "
Nov 02 14:26:05.665258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.665225    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-certs\") pod \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") "
Nov 02 14:26:05.665344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.665313    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-envoy\") pod \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") "
Nov 02 14:26:05.665419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.665337    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-socket" pod "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" (UID: "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:26:05.665419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.665363    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-kube-api-access-gcchz\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-kube-api-access-gcchz\") pod \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\" (UID: \"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec\") "
Nov 02 14:26:05.665588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.665567    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-credential-socket" pod "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" (UID: "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:26:05.665662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.665593    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-certs" pod "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" (UID: "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:26:05.665912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.665860    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istiod-ca-cert" pod "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" (UID: "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:05.666244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.666220    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:05.666356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.666248    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:05.666356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.666262    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:05.666356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.666278    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:05.666356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.666291    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:05.667378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.667330    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-podinfo" pod "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" (UID: "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:26:05.667793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.667762    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-envoy" pod "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" (UID: "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:26:05.667873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.667850    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-token" pod "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" (UID: "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:05.667945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.667925    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-kube-api-access-gcchz" pod "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" (UID: "3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec"). InnerVolumeSpecName "kube-api-access-gcchz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:05.767746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.767716    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:05.767746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.767741    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcchz\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-kube-api-access-gcchz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:05.767746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.767751    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:05.767994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:05.767762    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:06.189589 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:06.189555    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d13f4/agc-cred-3fbd4dc2e303: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:06.189589 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:06.189585    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:06.189881 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:06.189654    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials podName:4b9252e8-f794-42a1-a041-dbbb5b70f40f nodeName:}" failed. No retries permitted until 2026-11-02 14:26:08.189635227 +0000 UTC m=+2558.940939661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f") : secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:06.391124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.390757    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w" event={"ID":"8007ba35-5dca-48bc-8401-62896c8af6e8","Type":"ContainerStarted","Data":"eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346"}
Nov 02 14:26:06.391124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.390824    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w" event={"ID":"8007ba35-5dca-48bc-8401-62896c8af6e8","Type":"ContainerStarted","Data":"79acc9ff5eb3053f22f542fef05f704f983baf49f44eb7bb8b1cce14bd9f7197"}
Nov 02 14:26:06.391124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.390850    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:06.393308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.393284    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"
Nov 02 14:26:06.393584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.393326    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2" event={"ID":"3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec","Type":"ContainerDied","Data":"d1636d02b2398680600dbb1669419236795254907f14e0ddd5500b49702a3d57"}
Nov 02 14:26:06.393584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.393393    5034 scope.go:122] "RemoveContainer" containerID="4c9813146911ff714bfffb146239169043c41271742a8ada3e607337d544b99a"
Nov 02 14:26:06.440057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.440011    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w" podStartSLOduration=3.4399979370000002 podStartE2EDuration="3.439997937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:06.437209577 +0000 UTC m=+2557.188514014" watchObservedRunningTime="2026-11-02 14:26:06.439997937 +0000 UTC m=+2557.191302375"
Nov 02 14:26:06.487746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.487715    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"]
Nov 02 14:26:06.495459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.495428    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"]
Nov 02 14:26:06.496688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.496664    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy"
Nov 02 14:26:06.496688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.496688    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy"
Nov 02 14:26:06.497037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.497008    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" containerName="istio-proxy"
Nov 02 14:26:06.503533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.503512    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.506438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.506412    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-wkjd2"]
Nov 02 14:26:06.529622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.529431    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-b1ab1ce91238\""
Nov 02 14:26:06.530269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.530244    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"routing-overlay\""
Nov 02 14:26:06.530439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.530326    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-e2e--6a9d33b0-dockercfg-b6wf9\""
Nov 02 14:26:06.531402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.531382    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"agc-cred-821b4dcc9b92\""
Nov 02 14:26:06.536220 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.536194    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"]
Nov 02 14:26:06.596438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.596398    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.596640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.596466    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.596751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.596722    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.596883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.596854    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c29b\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-kube-api-access-2c29b\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.631726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.631702    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"openshift-service-ca.crt\""
Nov 02 14:26:06.631726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.631706    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ee3eca\""
Nov 02 14:26:06.631995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.631751    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ee3eca\"/\"kube-root-ca.crt\""
Nov 02 14:26:06.698535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.698445    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.698710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.698616    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.698710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.698688    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c29b\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-kube-api-access-2c29b\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.698999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.698975    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.699540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.699508    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.699759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.699736    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.701369 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.701342    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.723692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.723664    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c29b\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-kube-api-access-2c29b\") pod \"payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") " pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:06.823725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:06.823690    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:07.077509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.077469    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"]
Nov 02 14:26:07.084640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.084611    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.105816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.105370    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a02c35-openshift-default-dockercfg-bm9cg\""
Nov 02 14:26:07.109687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.109660    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"]
Nov 02 14:26:07.174496 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:07.159313    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d132505_73e8_4ba9_ac3e_a6c502c109db.slice/crio-a5eb647288cef411b26f4147ff0bf84e07532eb79f2e31e603657508cb102b01 WatchSource:0}: Error finding container a5eb647288cef411b26f4147ff0bf84e07532eb79f2e31e603657508cb102b01: Status 404 returned error can't find the container with id a5eb647288cef411b26f4147ff0bf84e07532eb79f2e31e603657508cb102b01
Nov 02 14:26:07.174496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.162034    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"]
Nov 02 14:26:07.209491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.209453    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-data\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.209698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.209670    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk5ff\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-kube-api-access-wk5ff\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.209795 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.209744    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-certs\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.209866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.209848    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-credential-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.209973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.209956    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istiod-ca-cert\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.210074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.210060    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-token\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.210138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.210086    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.210204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.210134    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-envoy\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.210204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.210163    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-podinfo\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.295284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.295251    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"]
Nov 02 14:26:07.300364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.300341    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.312036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.311800    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istiod-ca-cert\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.312036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.311948    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-token\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.312036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.311977    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.312303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.312041    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-envoy\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.312303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.312071    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-podinfo\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.312303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.312201    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-data\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.312472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.312331    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk5ff\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-kube-api-access-wk5ff\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.312472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.312387    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-certs\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.312472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.312444    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-credential-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.314377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.312917    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-credential-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.314377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.313599    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-data\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.314377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.313878    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-certs\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.314377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.314134    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istiod-ca-cert\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.314377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.314215    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-socket\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.317529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.317489    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2d13f4\"/\"e2e-cfl-4239ef-provider-cred\""
Nov 02 14:26:07.319736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.319691    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-envoy\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.322401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.322374    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"]
Nov 02 14:26:07.323181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.323143    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-podinfo\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.338358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.338255    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk5ff\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-kube-api-access-wk5ff\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.346908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.346854    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-token\") pod \"e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") " pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.407414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.407378    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec" path="/var/lib/kubelet/pods/3d97ee28-1ccf-4ce4-8c26-5c50a0ccf4ec/volumes"
Nov 02 14:26:07.413260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.413219    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:07.413949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.413912    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.417917 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.417886    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.418076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.418059    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.418166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.418121    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97" event={"ID":"1d132505-73e8-4ba9-ac3e-a6c502c109db","Type":"ContainerStarted","Data":"8e2b23b6da8d5010f6a30f36bc4dbeabc2d14ddbfa43e596f4463f1a71207756"}
Nov 02 14:26:07.418166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.418146    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhnp\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-kube-api-access-kfhnp\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.418285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.418171    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97" event={"ID":"1d132505-73e8-4ba9-ac3e-a6c502c109db","Type":"ContainerStarted","Data":"a5eb647288cef411b26f4147ff0bf84e07532eb79f2e31e603657508cb102b01"}
Nov 02 14:26:07.418285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.418218    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:07.464535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.464470    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97" podStartSLOduration=1.464451363 podStartE2EDuration="1.464451363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:07.46252817 +0000 UTC m=+2558.213832608" watchObservedRunningTime="2026-11-02 14:26:07.464451363 +0000 UTC m=+2558.215755803"
Nov 02 14:26:07.522518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.521067    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.522518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.521148    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.522518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.521637    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.522518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.521874    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhnp\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-kube-api-access-kfhnp\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.522518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.522236    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.523228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.522719    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.525838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.524463    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.547294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.547269    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhnp\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-kube-api-access-kfhnp\") pod \"payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") " pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.617436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.617145    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:07.781836 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:07.781803    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb27b4b1_3b93_415a_82b0_1a4cb112783c.slice/crio-e58c8393465defb16855a70e26855d0defc509f87f68a13b7594d0ed430e75ed WatchSource:0}: Error finding container e58c8393465defb16855a70e26855d0defc509f87f68a13b7594d0ed430e75ed: Status 404 returned error can't find the container with id e58c8393465defb16855a70e26855d0defc509f87f68a13b7594d0ed430e75ed
Nov 02 14:26:07.784581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.784538    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:07.784716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.784629    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:07.784716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.784677    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:07.786412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.786378    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"]
Nov 02 14:26:07.950428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:07.950396    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"]
Nov 02 14:26:07.954044 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:07.953997    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32758a5_836f_492a_aa2e_26653d3ca2a2.slice/crio-a3245cc1f7bbdebf95b0858084f3b2f0fbf8324eac98f22ecf3775b0681eb7cd WatchSource:0}: Error finding container a3245cc1f7bbdebf95b0858084f3b2f0fbf8324eac98f22ecf3775b0681eb7cd: Status 404 returned error can't find the container with id a3245cc1f7bbdebf95b0858084f3b2f0fbf8324eac98f22ecf3775b0681eb7cd
Nov 02 14:26:08.241217 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:08.241188    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d13f4/agc-cred-3fbd4dc2e303: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:08.241217 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:08.241212    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:08.241470 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:08.241278    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials podName:4b9252e8-f794-42a1-a041-dbbb5b70f40f nodeName:}" failed. No retries permitted until 2026-11-02 14:26:12.241259267 +0000 UTC m=+2562.992563682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f") : secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:08.430096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:08.430057    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s" event={"ID":"b32758a5-836f-492a-aa2e-26653d3ca2a2","Type":"ContainerStarted","Data":"511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2"}
Nov 02 14:26:08.430293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:08.430114    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s" event={"ID":"b32758a5-836f-492a-aa2e-26653d3ca2a2","Type":"ContainerStarted","Data":"a3245cc1f7bbdebf95b0858084f3b2f0fbf8324eac98f22ecf3775b0681eb7cd"}
Nov 02 14:26:08.430293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:08.430143    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:08.432434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:08.432404    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" event={"ID":"bb27b4b1-3b93-415a-82b0-1a4cb112783c","Type":"ContainerStarted","Data":"c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca"}
Nov 02 14:26:08.432539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:08.432457    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" event={"ID":"bb27b4b1-3b93-415a-82b0-1a4cb112783c","Type":"ContainerStarted","Data":"e58c8393465defb16855a70e26855d0defc509f87f68a13b7594d0ed430e75ed"}
Nov 02 14:26:08.478715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:08.478656    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s" podStartSLOduration=1.478639616 podStartE2EDuration="1.478639616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:08.476362089 +0000 UTC m=+2559.227666529" watchObservedRunningTime="2026-11-02 14:26:08.478639616 +0000 UTC m=+2559.229944054"
Nov 02 14:26:08.528362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:08.528271    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" podStartSLOduration=1.528257781 podStartE2EDuration="1.528257781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:08.52611199 +0000 UTC m=+2559.277416430" watchObservedRunningTime="2026-11-02 14:26:08.528257781 +0000 UTC m=+2559.279562215"
Nov 02 14:26:09.416253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:09.415832    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Nov 02 14:26:09.416253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:09.415900    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Nov 02 14:26:09.425468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:09.425440    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:10.418068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:10.417975    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Nov 02 14:26:10.420319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:10.418069    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Nov 02 14:26:10.704914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:10.692047    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": read tcp 10.132.0.2:44594->10.132.1.22:15021: read: connection reset by peer" start-of-body=
Nov 02 14:26:10.704914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:10.692128    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": read tcp 10.132.0.2:44594->10.132.1.22:15021: read: connection reset by peer"
Nov 02 14:26:11.414888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:11.414851    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Nov 02 14:26:11.415088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:11.414910    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Nov 02 14:26:11.509466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:11.509422    5034 generic.go:356] "Generic (PLEG): container finished" podID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerID="011a8d5633394df0e612db47dd7fe4c68859667ca1e4c756135c79f580b17a6e" exitCode=0
Nov 02 14:26:11.509901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:11.509487    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" event={"ID":"23cc2e99-f558-4e2c-be3a-ed110bebf2e9","Type":"ContainerDied","Data":"011a8d5633394df0e612db47dd7fe4c68859667ca1e4c756135c79f580b17a6e"}
Nov 02 14:26:11.509999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:11.509981    5034 scope.go:122] "RemoveContainer" containerID="011a8d5633394df0e612db47dd7fe4c68859667ca1e4c756135c79f580b17a6e"
Nov 02 14:26:11.512054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:11.512021    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:11.512213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:11.512110    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:11.512213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:11.512163    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:12.245472 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:12.245292    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d13f4/agc-cred-3fbd4dc2e303: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:12.245472 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:12.245322    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:12.245472 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:12.245369    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials podName:4b9252e8-f794-42a1-a041-dbbb5b70f40f nodeName:}" failed. No retries permitted until 2026-11-02 14:26:20.245356014 +0000 UTC m=+2570.996660431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f") : secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:12.424804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:12.415075    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Nov 02 14:26:12.424804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:12.415133    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Nov 02 14:26:12.541247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:12.541165    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" event={"ID":"23cc2e99-f558-4e2c-be3a-ed110bebf2e9","Type":"ContainerStarted","Data":"f1089380c61b5aefd94ea3c5091a65b74f6e9a8b593b069578e2dc26144c903c"}
Nov 02 14:26:13.031839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:13.031500    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Nov 02 14:26:13.031839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:13.031568    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Nov 02 14:26:13.032092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:13.031871    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:26:13.429845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:13.429258    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Nov 02 14:26:13.429845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:13.429314    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Nov 02 14:26:14.024944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:14.024899    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Nov 02 14:26:14.025472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:14.025004    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Nov 02 14:26:14.415325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:14.415241    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body=
Nov 02 14:26:14.415596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:14.415568    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused"
Nov 02 14:26:15.024195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:15.024159    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Nov 02 14:26:15.024478 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:15.024220    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Nov 02 14:26:15.410444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:15.410371    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:15.419431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:15.419405    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:15.419669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:15.419641    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:15.420348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:15.420329    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:26:16.024138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:16.024096    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Nov 02 14:26:16.024323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:16.024157    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Nov 02 14:26:17.024880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:17.024847    5034 patch_prober.go:29] interesting pod/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused" start-of-body=
Nov 02 14:26:17.025431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:17.024908    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.22:15021/healthz/ready\": dial tcp 10.132.1.22:15021: connect: connection refused"
Nov 02 14:26:17.421122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:17.421043    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:26:17.513808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:17.513685    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"]
Nov 02 14:26:18.028894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:18.028863    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:26:18.029385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:18.029253    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:26:18.030392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:18.030372    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:26:18.416921 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:18.416829    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:26:19.443937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:19.443909    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:26:19.444378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:19.443979    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:19.595267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:19.594996    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"]
Nov 02 14:26:20.277339 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:20.277294    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2d13f4/agc-cred-3fbd4dc2e303: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:20.277339 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:20.277335    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb: secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:20.277597 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:20.277413    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials podName:4b9252e8-f794-42a1-a041-dbbb5b70f40f nodeName:}" failed. No retries permitted until 2026-11-02 14:26:36.277392879 +0000 UTC m=+2587.028697309 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f") : secret "agc-cred-3fbd4dc2e303" not found
Nov 02 14:26:21.447425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.447386    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"]
Nov 02 14:26:21.454094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.454066    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.483858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.483830    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"]
Nov 02 14:26:21.487423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.487393    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-27gvt\""
Nov 02 14:26:21.494586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.494550    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.494741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.494680    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.494862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.494764    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.494937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.494902    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.494997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.494956    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2dnl\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-kube-api-access-m2dnl\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.494997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.494986    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.495124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.495071    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.495124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.495111    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.495342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.495319    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.597137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.597102    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.597344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.597287    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.597426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.597365    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.597485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.597466    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.597560 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.597525    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.597624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.597604    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.597683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.597639    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2dnl\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-kube-api-access-m2dnl\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.597683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.597669    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.597857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.597736    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.598246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.598213    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.598674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.598627    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.598674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.598663    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.598964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.598944    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.599047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.599029    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.600373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.600345    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.600632 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.600616    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.624927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.624899    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.625083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.625067    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2dnl\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-kube-api-access-m2dnl\") pod \"maas-default-gateway-openshift-default-6886b96f9-8rxhv\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:21.768966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:21.768932    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:22.099806 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:22.099752    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2833a5ee_1c7b_4bbb_9a5b_e1ea68e8d920.slice/crio-c9e647bdc0a2526221bf7fd9526fa62b8d802ac10226ec7f2ae602bcb97b0883 WatchSource:0}: Error finding container c9e647bdc0a2526221bf7fd9526fa62b8d802ac10226ec7f2ae602bcb97b0883: Status 404 returned error can't find the container with id c9e647bdc0a2526221bf7fd9526fa62b8d802ac10226ec7f2ae602bcb97b0883
Nov 02 14:26:22.101927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.101898    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"]
Nov 02 14:26:22.102846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.102807    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:22.102938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.102881    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:22.102938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.102911    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:22.514127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.514072    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb" podUID="8c60ad12-034c-4362-bba6-d3e2ff33caf8" containerName="payload-processing" containerID="cri-o://835c9f4026aaf264fa001d5b87409e45dd127abf62075e14c2b46748ad8aa1a3" gracePeriod=25
Nov 02 14:26:22.688204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.688163    5034 generic.go:356] "Generic (PLEG): container finished" podID="8c60ad12-034c-4362-bba6-d3e2ff33caf8" containerID="835c9f4026aaf264fa001d5b87409e45dd127abf62075e14c2b46748ad8aa1a3" exitCode=0
Nov 02 14:26:22.688383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.688233    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb" event={"ID":"8c60ad12-034c-4362-bba6-d3e2ff33caf8","Type":"ContainerDied","Data":"835c9f4026aaf264fa001d5b87409e45dd127abf62075e14c2b46748ad8aa1a3"}
Nov 02 14:26:22.690582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.690552    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" event={"ID":"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920","Type":"ContainerStarted","Data":"eb05fabc6444e265685056cf0dad1fd2bf1db0a8c63c636efccdf73b112371ae"}
Nov 02 14:26:22.690702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.690609    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" event={"ID":"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920","Type":"ContainerStarted","Data":"c9e647bdc0a2526221bf7fd9526fa62b8d802ac10226ec7f2ae602bcb97b0883"}
Nov 02 14:26:22.753379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.753326    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podStartSLOduration=1.75331358 podStartE2EDuration="1.75331358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:22.751661763 +0000 UTC m=+2573.502966214" watchObservedRunningTime="2026-11-02 14:26:22.75331358 +0000 UTC m=+2573.504618018"
Nov 02 14:26:22.769407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.769315    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:22.829719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.829687    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:26:22.923803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.923760    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-kube-api-access-tm64w\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-kube-api-access-tm64w\") pod \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") "
Nov 02 14:26:22.923991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.923878    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-provider-credentials\") pod \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") "
Nov 02 14:26:22.923991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.923931    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-routing-overlay-volume\") pod \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") "
Nov 02 14:26:22.924122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.924108    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-plugins-config-volume\") pod \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\" (UID: \"8c60ad12-034c-4362-bba6-d3e2ff33caf8\") "
Nov 02 14:26:22.924407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.924357    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-routing-overlay-volume" pod "8c60ad12-034c-4362-bba6-d3e2ff33caf8" (UID: "8c60ad12-034c-4362-bba6-d3e2ff33caf8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:22.924570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.924500    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-plugins-config-volume" pod "8c60ad12-034c-4362-bba6-d3e2ff33caf8" (UID: "8c60ad12-034c-4362-bba6-d3e2ff33caf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:22.924874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.924855    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:22.924929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.924877    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c60ad12-034c-4362-bba6-d3e2ff33caf8-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:22.926588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.926560    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-kube-api-access-tm64w" pod "8c60ad12-034c-4362-bba6-d3e2ff33caf8" (UID: "8c60ad12-034c-4362-bba6-d3e2ff33caf8"). InnerVolumeSpecName "kube-api-access-tm64w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:22.926670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:22.926626    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-provider-credentials" pod "8c60ad12-034c-4362-bba6-d3e2ff33caf8" (UID: "8c60ad12-034c-4362-bba6-d3e2ff33caf8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:23.026394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:23.026315    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm64w\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-kube-api-access-tm64w\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:23.026394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:23.026342    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c60ad12-034c-4362-bba6-d3e2ff33caf8-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:23.708972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:23.708751    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"
Nov 02 14:26:23.708972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:23.708757    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb" event={"ID":"8c60ad12-034c-4362-bba6-d3e2ff33caf8","Type":"ContainerDied","Data":"c968a2bec73950d3d75706dd3700ac823b720c201d909164b8fff04f40ebe52f"}
Nov 02 14:26:23.708972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:23.708957    5034 scope.go:122] "RemoveContainer" containerID="835c9f4026aaf264fa001d5b87409e45dd127abf62075e14c2b46748ad8aa1a3"
Nov 02 14:26:23.788802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:23.774378    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 14:26:23.788802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:23.774433    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 14:26:23.810010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:23.809968    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"]
Nov 02 14:26:23.816999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:23.816968    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-6cc5jxknb"]
Nov 02 14:26:24.601801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:24.601480    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" podUID="4b9252e8-f794-42a1-a041-dbbb5b70f40f" containerName="payload-processing" containerID="cri-o://48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27" gracePeriod=25
Nov 02 14:26:24.790804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:24.780066    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 14:26:24.790804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:24.780125    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 14:26:24.994801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:24.994176    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:25.063479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.063448    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials\") pod \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") "
Nov 02 14:26:25.063641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.063577    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-plugins-config-volume\") pod \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") "
Nov 02 14:26:25.063641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.063623    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-routing-overlay-volume\") pod \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") "
Nov 02 14:26:25.063725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.063641    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-kube-api-access-m9p46\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-kube-api-access-m9p46\") pod \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\" (UID: \"4b9252e8-f794-42a1-a041-dbbb5b70f40f\") "
Nov 02 14:26:25.064640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.064608    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-plugins-config-volume" pod "4b9252e8-f794-42a1-a041-dbbb5b70f40f" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:25.064925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.064863    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-routing-overlay-volume" pod "4b9252e8-f794-42a1-a041-dbbb5b70f40f" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:25.065217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.065194    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:25.065319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.065221    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9252e8-f794-42a1-a041-dbbb5b70f40f-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:25.066329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.066300    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials" pod "4b9252e8-f794-42a1-a041-dbbb5b70f40f" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:25.067221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.067198    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-kube-api-access-m9p46" pod "4b9252e8-f794-42a1-a041-dbbb5b70f40f" (UID: "4b9252e8-f794-42a1-a041-dbbb5b70f40f"). InnerVolumeSpecName "kube-api-access-m9p46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:25.166544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.166503    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:25.166544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.166531    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9p46\" (UniqueName: \"kubernetes.io/projected/4b9252e8-f794-42a1-a041-dbbb5b70f40f-kube-api-access-m9p46\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:25.412008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.411980    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c60ad12-034c-4362-bba6-d3e2ff33caf8" path="/var/lib/kubelet/pods/8c60ad12-034c-4362-bba6-d3e2ff33caf8/volumes"
Nov 02 14:26:25.733766 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.733739    5034 generic.go:356] "Generic (PLEG): container finished" podID="4b9252e8-f794-42a1-a041-dbbb5b70f40f" containerID="48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27" exitCode=0
Nov 02 14:26:25.734011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.733801    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" event={"ID":"4b9252e8-f794-42a1-a041-dbbb5b70f40f","Type":"ContainerDied","Data":"48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27"}
Nov 02 14:26:25.734121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.734108    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb" event={"ID":"4b9252e8-f794-42a1-a041-dbbb5b70f40f","Type":"ContainerDied","Data":"edea2a50318ca8b649529c939e9bf79389264f7688b557c552e299d24a09a1a3"}
Nov 02 14:26:25.734192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.733829    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"
Nov 02 14:26:25.734330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.734146    5034 scope.go:122] "RemoveContainer" containerID="48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27"
Nov 02 14:26:25.767801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.765130    5034 scope.go:122] "RemoveContainer" containerID="48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27"
Nov 02 14:26:25.767801 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:25.765710    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27\": container with ID starting with 48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27 not found: ID does not exist" containerID="48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27"
Nov 02 14:26:25.767801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.765994    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27"} err="failed to get container status \"48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27\": rpc error: code = NotFound desc = could not find container \"48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27\": container with ID starting with 48d1268b98b3ef8ca8c29f3aabff5ba59ec16ec4a7150ad064eabfb5c1f1ea27 not found: ID does not exist"
Nov 02 14:26:25.787801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.775109    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 14:26:25.787801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.775161    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 14:26:25.812799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.812099    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"]
Nov 02 14:26:25.819718 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:25.819621    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-765f6jvhb"]
Nov 02 14:26:26.776804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:26.773121    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 14:26:26.776804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:26.773181    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 14:26:27.386856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.386818    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"]
Nov 02 14:26:27.388793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.388752    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4b9252e8-f794-42a1-a041-dbbb5b70f40f" containerName="payload-processing"
Nov 02 14:26:27.388974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.388959    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9252e8-f794-42a1-a041-dbbb5b70f40f" containerName="payload-processing"
Nov 02 14:26:27.389082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.389072    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8c60ad12-034c-4362-bba6-d3e2ff33caf8" containerName="payload-processing"
Nov 02 14:26:27.389168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.389158    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c60ad12-034c-4362-bba6-d3e2ff33caf8" containerName="payload-processing"
Nov 02 14:26:27.389450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.389432    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c60ad12-034c-4362-bba6-d3e2ff33caf8" containerName="payload-processing"
Nov 02 14:26:27.389591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.389579    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9252e8-f794-42a1-a041-dbbb5b70f40f" containerName="payload-processing"
Nov 02 14:26:27.405958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.405929    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.426007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.425984    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-4c2811-openshift-default-dockercfg-pqrrz\""
Nov 02 14:26:27.432022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.431993    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9252e8-f794-42a1-a041-dbbb5b70f40f" path="/var/lib/kubelet/pods/4b9252e8-f794-42a1-a041-dbbb5b70f40f/volumes"
Nov 02 14:26:27.432720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.432696    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"]
Nov 02 14:26:27.537427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.536506    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.537427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.536600    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.537427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.536692    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.537427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.536739    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.537427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.536757    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.537427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.536929    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.537427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.536968    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppmc\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-kube-api-access-kppmc\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.537427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.537061    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.537427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.537091    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.638609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.638514    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.638609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.638599    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.638896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.638628    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.638896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.638701    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.638896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.638732    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppmc\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-kube-api-access-kppmc\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.638896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.638755    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.638896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.638799    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.639179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.638970    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.639179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.639055    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.639375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.639346    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istiod-ca-cert\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.639497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.639468    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-data\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.639698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.639677    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-credential-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.639898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.639875    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-certs\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.639966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.639889    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-socket\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.641743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.641718    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-envoy\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.641947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.641930    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-podinfo\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.668244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.668218    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-token\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.668362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.668297    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppmc\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-kube-api-access-kppmc\") pod \"e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") " pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.732909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.732879    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:27.775819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.769384    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 14:26:27.775819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:27.769439    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 14:26:28.208802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:28.206699    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"]
Nov 02 14:26:28.337725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:28.337576    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:28.337725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:28.337669    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:28.337725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:28.337715    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:28.764808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:28.764757    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" event={"ID":"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1","Type":"ContainerStarted","Data":"776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f"}
Nov 02 14:26:28.765209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:28.764835    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" event={"ID":"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1","Type":"ContainerStarted","Data":"e3b18142527ba1355c857e4ff4ed4fc114f6dfe9cf1aa3afaaea28fe159e259a"}
Nov 02 14:26:28.776802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:28.770116    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 14:26:28.776802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:28.770180    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 14:26:28.831939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:28.831886    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" podStartSLOduration=1.831872818 podStartE2EDuration="1.831872818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:28.83129083 +0000 UTC m=+2579.582595281" watchObservedRunningTime="2026-11-02 14:26:28.831872818 +0000 UTC m=+2579.583177258"
Nov 02 14:26:29.772301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:29.772261    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 14:26:29.772973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:29.772938    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 14:26:29.793864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:29.793810    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:29.794816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:29.794504    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 02 14:26:29.794816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:29.794576    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 02 14:26:30.733384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:30.733351    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 02 14:26:30.733640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:30.733607    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 02 14:26:30.773915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:30.773887    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8rxhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body=
Nov 02 14:26:30.774511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:30.774481    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused"
Nov 02 14:26:31.734158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.734120    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 02 14:26:31.734340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.734187    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 02 14:26:31.823478 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.823444    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"]
Nov 02 14:26:31.834997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.833768    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.839195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.839162    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"]
Nov 02 14:26:31.853657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.851371    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 14:26:31.853657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.851374    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 14:26:31.853657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.851428    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 14:26:31.853657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.851500    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5wglw\""
Nov 02 14:26:31.858683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.858647    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.858850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.858720    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/721d9747-d92b-47fc-9f0d-66e9d7a47db8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.858850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.858753    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.858850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.858800    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.859077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.858959    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4ldw\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-kube-api-access-z4ldw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.859077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.859045    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.859218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.859200    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.960649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.960619    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4ldw\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-kube-api-access-z4ldw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.961414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.961386    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.962025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.962004    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.962940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.962919    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.963209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.963189    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/721d9747-d92b-47fc-9f0d-66e9d7a47db8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.967261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.967123    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.968280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.967919    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.968280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.967837    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:31.968280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:31.966134    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:32.027410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.027281    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:32.034194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.034171    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/721d9747-d92b-47fc-9f0d-66e9d7a47db8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:32.037396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.037323    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4ldw\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-kube-api-access-z4ldw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:32.040550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.040500    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:32.040550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.040547    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-m2ls7\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:32.237068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.236760    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:32.237068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.236760    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:32.237450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.237425    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:32.238284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.238250    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:26:32.616897 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:32.616864    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721d9747_d92b_47fc_9f0d_66e9d7a47db8.slice/crio-ea502c5ce74bac8e30be1c2e1f7d6c773126a586f1e33bec9322e88b11d19700 WatchSource:0}: Error finding container ea502c5ce74bac8e30be1c2e1f7d6c773126a586f1e33bec9322e88b11d19700: Status 404 returned error can't find the container with id ea502c5ce74bac8e30be1c2e1f7d6c773126a586f1e33bec9322e88b11d19700
Nov 02 14:26:32.620344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.620312    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"]
Nov 02 14:26:32.620751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.620720    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:32.620862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.620837    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:26:32.733522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.733489    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 02 14:26:32.733670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.733549    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 02 14:26:32.928419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.928333    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7" event={"ID":"721d9747-d92b-47fc-9f0d-66e9d7a47db8","Type":"ContainerStarted","Data":"b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73"}
Nov 02 14:26:32.929031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.929010    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:32.929204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:32.929187    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7" event={"ID":"721d9747-d92b-47fc-9f0d-66e9d7a47db8","Type":"ContainerStarted","Data":"ea502c5ce74bac8e30be1c2e1f7d6c773126a586f1e33bec9322e88b11d19700"}
Nov 02 14:26:33.017084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:33.016213    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7" podStartSLOduration=2.01619439 podStartE2EDuration="2.01619439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:32.992866334 +0000 UTC m=+2583.744170774" watchObservedRunningTime="2026-11-02 14:26:33.01619439 +0000 UTC m=+2583.767498829"
Nov 02 14:26:33.733520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:33.733480    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 02 14:26:33.733734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:33.733543    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 02 14:26:33.942282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:33.942246    5034 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-m2ls7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:26:33.942888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:33.942856    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7" podUID="721d9747-d92b-47fc-9f0d-66e9d7a47db8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:26:34.733712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:34.733674    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body=
Nov 02 14:26:34.733950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:34.733738    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused"
Nov 02 14:26:34.951605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:34.951570    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:26:35.907847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:35.906998    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:35.908115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:35.908098    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:35.912448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:35.909298    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:26:36.825373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:36.825339    5034 scope.go:122] "RemoveContainer" containerID="f9f0a528d4f1134316acbdf183f30cbafd4003cdee02b68be9e6557e32cd3ee6"
Nov 02 14:26:36.837251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:36.837225    5034 scope.go:122] "RemoveContainer" containerID="a94e2b7937d0e0b06c941c2692dee37efa14b6ce160c9f1dc3f5413cf6039160"
Nov 02 14:26:36.847810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:36.847755    5034 scope.go:122] "RemoveContainer" containerID="c477f0e5db60f88644b2c7dec1fbc0199df1425d106b3f063d02b117643eaba4"
Nov 02 14:26:36.858072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:36.858051    5034 scope.go:122] "RemoveContainer" containerID="ee5b239c6a2d57d14c2fd6a5da21416276f0be0f90628885c3002fb28527432d"
Nov 02 14:26:37.363096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.363066    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"]
Nov 02 14:26:37.374627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.374593    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.396223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.396184    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:26:37.396591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.396301    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:26:37.396591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.396364    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:26:37.412825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.412278    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"]
Nov 02 14:26:37.412983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.412925    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:26:37.487572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.487311    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m7vq\" (UniqueName: \"kubernetes.io/projected/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-kube-api-access-7m7vq\") pod \"maas-api-revoke-sub-bec254d5-hmbcc\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.487771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.487598    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bec254d5-hmbcc\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.487771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.487705    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bec254d5-hmbcc\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.589046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.589003    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bec254d5-hmbcc\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.589258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.589209    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m7vq\" (UniqueName: \"kubernetes.io/projected/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-kube-api-access-7m7vq\") pod \"maas-api-revoke-sub-bec254d5-hmbcc\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.589397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.589378    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bec254d5-hmbcc\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.589747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.589721    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bec254d5-hmbcc\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.591854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.591828    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bec254d5-hmbcc\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.616654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.616595    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m7vq\" (UniqueName: \"kubernetes.io/projected/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-kube-api-access-7m7vq\") pod \"maas-api-revoke-sub-bec254d5-hmbcc\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:37.717973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:37.717940    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:38.049667 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:38.049636    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4cc27d_dbf7_4596_98f6_d2efc5ce6891.slice/crio-156c8579e2ab581b9fc1e70103da94d1eaff8f60495c1e25bca964bc3898c107 WatchSource:0}: Error finding container 156c8579e2ab581b9fc1e70103da94d1eaff8f60495c1e25bca964bc3898c107: Status 404 returned error can't find the container with id 156c8579e2ab581b9fc1e70103da94d1eaff8f60495c1e25bca964bc3898c107
Nov 02 14:26:38.052389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:38.052364    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"]
Nov 02 14:26:39.032978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:39.027598    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc" event={"ID":"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891","Type":"ContainerStarted","Data":"d710f3adf4e31da8445305c6005848bdd34daba2baf7c4eacf95b6ff7188d8c7"}
Nov 02 14:26:39.032978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:39.027665    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc" event={"ID":"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891","Type":"ContainerStarted","Data":"156c8579e2ab581b9fc1e70103da94d1eaff8f60495c1e25bca964bc3898c107"}
Nov 02 14:26:39.084230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:39.082746    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc" podStartSLOduration=2.082727534 podStartE2EDuration="2.082727534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:39.080526752 +0000 UTC m=+2589.831831193" watchObservedRunningTime="2026-11-02 14:26:39.082727534 +0000 UTC m=+2589.834031980"
Nov 02 14:26:40.038261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:40.038223    5034 generic.go:356] "Generic (PLEG): container finished" podID="cf4cc27d-dbf7-4596-98f6-d2efc5ce6891" containerID="d710f3adf4e31da8445305c6005848bdd34daba2baf7c4eacf95b6ff7188d8c7" exitCode=0
Nov 02 14:26:40.038458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:40.038312    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc" event={"ID":"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891","Type":"ContainerDied","Data":"d710f3adf4e31da8445305c6005848bdd34daba2baf7c4eacf95b6ff7188d8c7"}
Nov 02 14:26:40.964163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:40.964130    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"]
Nov 02 14:26:41.246200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.246176    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:41.347888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.347853    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-xks-ca\") pod \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") "
Nov 02 14:26:41.348163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.348145    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-kube-api-access-7m7vq\" (UniqueName: \"kubernetes.io/projected/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-kube-api-access-7m7vq\") pod \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") "
Nov 02 14:26:41.348263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.348248    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-openshift-ca\") pod \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\" (UID: \"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891\") "
Nov 02 14:26:41.349292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.349255    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-openshift-ca" pod "cf4cc27d-dbf7-4596-98f6-d2efc5ce6891" (UID: "cf4cc27d-dbf7-4596-98f6-d2efc5ce6891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:41.349481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.349448    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:41.350866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.350768    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-xks-ca" pod "cf4cc27d-dbf7-4596-98f6-d2efc5ce6891" (UID: "cf4cc27d-dbf7-4596-98f6-d2efc5ce6891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:26:41.352336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.352301    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-kube-api-access-7m7vq" pod "cf4cc27d-dbf7-4596-98f6-d2efc5ce6891" (UID: "cf4cc27d-dbf7-4596-98f6-d2efc5ce6891"). InnerVolumeSpecName "kube-api-access-7m7vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:41.450936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.450905    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:41.451083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:41.450948    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m7vq\" (UniqueName: \"kubernetes.io/projected/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891-kube-api-access-7m7vq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:42.057946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:42.057919    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"
Nov 02 14:26:42.058380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:42.057961    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc" event={"ID":"cf4cc27d-dbf7-4596-98f6-d2efc5ce6891","Type":"ContainerDied","Data":"156c8579e2ab581b9fc1e70103da94d1eaff8f60495c1e25bca964bc3898c107"}
Nov 02 14:26:42.058380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:42.058056    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="156c8579e2ab581b9fc1e70103da94d1eaff8f60495c1e25bca964bc3898c107"
Nov 02 14:26:45.829399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.829366    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"]
Nov 02 14:26:45.830241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.830225    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cf4cc27d-dbf7-4596-98f6-d2efc5ce6891" containerName="revoke-keys"
Nov 02 14:26:45.830304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.830244    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4cc27d-dbf7-4596-98f6-d2efc5ce6891" containerName="revoke-keys"
Nov 02 14:26:45.830422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.830413    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4cc27d-dbf7-4596-98f6-d2efc5ce6891" containerName="revoke-keys"
Nov 02 14:26:45.834939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.834917    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:45.856604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.856579    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"]
Nov 02 14:26:45.865066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.865038    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:26:45.865066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.865045    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:26:45.867649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.867621    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:26:45.867755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.867723    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:26:45.926082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.926051    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vch6s\" (UniqueName: \"kubernetes.io/projected/296e26a3-3f71-4f75-89b9-4592599d94a3-kube-api-access-vch6s\") pod \"maas-api-revoke-sub-4abe2db4-nhtzz\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:45.926260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.926135    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4abe2db4-nhtzz\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:45.926260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.926203    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4abe2db4-nhtzz\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:45.965467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:45.965424    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97" podUID="1d132505-73e8-4ba9-ac3e-a6c502c109db" containerName="payload-processing" containerID="cri-o://8e2b23b6da8d5010f6a30f36bc4dbeabc2d14ddbfa43e596f4463f1a71207756" gracePeriod=25
Nov 02 14:26:46.034832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.034086    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vch6s\" (UniqueName: \"kubernetes.io/projected/296e26a3-3f71-4f75-89b9-4592599d94a3-kube-api-access-vch6s\") pod \"maas-api-revoke-sub-4abe2db4-nhtzz\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:46.034832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.034205    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4abe2db4-nhtzz\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:46.034832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.034269    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4abe2db4-nhtzz\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:46.037210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.036623    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4abe2db4-nhtzz\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:46.045333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.045293    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4abe2db4-nhtzz\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:46.061945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.061917    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vch6s\" (UniqueName: \"kubernetes.io/projected/296e26a3-3f71-4f75-89b9-4592599d94a3-kube-api-access-vch6s\") pod \"maas-api-revoke-sub-4abe2db4-nhtzz\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:46.120404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.120371    5034 generic.go:356] "Generic (PLEG): container finished" podID="1d132505-73e8-4ba9-ac3e-a6c502c109db" containerID="8e2b23b6da8d5010f6a30f36bc4dbeabc2d14ddbfa43e596f4463f1a71207756" exitCode=0
Nov 02 14:26:46.120593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.120500    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97" event={"ID":"1d132505-73e8-4ba9-ac3e-a6c502c109db","Type":"ContainerDied","Data":"8e2b23b6da8d5010f6a30f36bc4dbeabc2d14ddbfa43e596f4463f1a71207756"}
Nov 02 14:26:46.150451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.150408    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:46.292087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.292055    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:46.444120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.444050    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-provider-credentials\") pod \"1d132505-73e8-4ba9-ac3e-a6c502c109db\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") "
Nov 02 14:26:46.444274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.444235    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-routing-overlay-volume\") pod \"1d132505-73e8-4ba9-ac3e-a6c502c109db\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") "
Nov 02 14:26:46.444403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.444387    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-kube-api-access-2c29b\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-kube-api-access-2c29b\") pod \"1d132505-73e8-4ba9-ac3e-a6c502c109db\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") "
Nov 02 14:26:46.444456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.444426    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-plugins-config-volume\") pod \"1d132505-73e8-4ba9-ac3e-a6c502c109db\" (UID: \"1d132505-73e8-4ba9-ac3e-a6c502c109db\") "
Nov 02 14:26:46.444639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.444615    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-routing-overlay-volume" pod "1d132505-73e8-4ba9-ac3e-a6c502c109db" (UID: "1d132505-73e8-4ba9-ac3e-a6c502c109db"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:46.444836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.444812    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-plugins-config-volume" pod "1d132505-73e8-4ba9-ac3e-a6c502c109db" (UID: "1d132505-73e8-4ba9-ac3e-a6c502c109db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:46.445259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.445234    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:46.445314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.445269    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d132505-73e8-4ba9-ac3e-a6c502c109db-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:46.446215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.446198    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-provider-credentials" pod "1d132505-73e8-4ba9-ac3e-a6c502c109db" (UID: "1d132505-73e8-4ba9-ac3e-a6c502c109db"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:46.446530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.446507    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-kube-api-access-2c29b" pod "1d132505-73e8-4ba9-ac3e-a6c502c109db" (UID: "1d132505-73e8-4ba9-ac3e-a6c502c109db"). InnerVolumeSpecName "kube-api-access-2c29b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:46.481494 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:46.481461    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296e26a3_3f71_4f75_89b9_4592599d94a3.slice/crio-8465850f31589ff3d1459a6c917678bfe14b2b9c51778fd9f3f84f7803ebb744 WatchSource:0}: Error finding container 8465850f31589ff3d1459a6c917678bfe14b2b9c51778fd9f3f84f7803ebb744: Status 404 returned error can't find the container with id 8465850f31589ff3d1459a6c917678bfe14b2b9c51778fd9f3f84f7803ebb744
Nov 02 14:26:46.483403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.483371    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"]
Nov 02 14:26:46.546899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.546875    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:46.546899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:46.546900    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c29b\" (UniqueName: \"kubernetes.io/projected/1d132505-73e8-4ba9-ac3e-a6c502c109db-kube-api-access-2c29b\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:47.131724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:47.131638    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz" event={"ID":"296e26a3-3f71-4f75-89b9-4592599d94a3","Type":"ContainerStarted","Data":"32a27f1e17b312764fef70b93c75e5a3e46af473bcd15128063525d03d835d27"}
Nov 02 14:26:47.131724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:47.131687    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz" event={"ID":"296e26a3-3f71-4f75-89b9-4592599d94a3","Type":"ContainerStarted","Data":"8465850f31589ff3d1459a6c917678bfe14b2b9c51778fd9f3f84f7803ebb744"}
Nov 02 14:26:47.133319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:47.133296    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"
Nov 02 14:26:47.133440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:47.133343    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97" event={"ID":"1d132505-73e8-4ba9-ac3e-a6c502c109db","Type":"ContainerDied","Data":"a5eb647288cef411b26f4147ff0bf84e07532eb79f2e31e603657508cb102b01"}
Nov 02 14:26:47.133440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:47.133404    5034 scope.go:122] "RemoveContainer" containerID="8e2b23b6da8d5010f6a30f36bc4dbeabc2d14ddbfa43e596f4463f1a71207756"
Nov 02 14:26:47.178219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:47.178172    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz" podStartSLOduration=2.178159006 podStartE2EDuration="2.178159006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:26:47.175330976 +0000 UTC m=+2597.926635416" watchObservedRunningTime="2026-11-02 14:26:47.178159006 +0000 UTC m=+2597.929463444"
Nov 02 14:26:47.220520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:47.220493    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"]
Nov 02 14:26:47.224667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:47.224644    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ee3eca/payload-processing-external-model-e2e-extmod-b-ee3eca-5bffpnn97"]
Nov 02 14:26:47.402513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:47.402437    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d132505-73e8-4ba9-ac3e-a6c502c109db" path="/var/lib/kubelet/pods/1d132505-73e8-4ba9-ac3e-a6c502c109db/volumes"
Nov 02 14:26:48.145418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.145384    5034 generic.go:356] "Generic (PLEG): container finished" podID="296e26a3-3f71-4f75-89b9-4592599d94a3" containerID="32a27f1e17b312764fef70b93c75e5a3e46af473bcd15128063525d03d835d27" exitCode=0
Nov 02 14:26:48.145917 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.145480    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz" event={"ID":"296e26a3-3f71-4f75-89b9-4592599d94a3","Type":"ContainerDied","Data":"32a27f1e17b312764fef70b93c75e5a3e46af473bcd15128063525d03d835d27"}
Nov 02 14:26:48.292387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.292352    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"]
Nov 02 14:26:48.293409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.293385    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1d132505-73e8-4ba9-ac3e-a6c502c109db" containerName="payload-processing"
Nov 02 14:26:48.293409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.293408    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d132505-73e8-4ba9-ac3e-a6c502c109db" containerName="payload-processing"
Nov 02 14:26:48.293616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.293583    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d132505-73e8-4ba9-ac3e-a6c502c109db" containerName="payload-processing"
Nov 02 14:26:48.297936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.297914    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.317699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.317676    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"]
Nov 02 14:26:48.382889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.382856    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02c2f7d-4w7wq\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.383202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.383175    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbfcm\" (UniqueName: \"kubernetes.io/projected/1c367b83-3095-48d8-9a05-a481fc5bc516-kube-api-access-fbfcm\") pod \"maas-api-revoke-sub-e02c2f7d-4w7wq\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.383302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.383280    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02c2f7d-4w7wq\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.415367 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:48.415287    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:26:48.485040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.485008    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02c2f7d-4w7wq\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.485249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.485123    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02c2f7d-4w7wq\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.485460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.485438    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbfcm\" (UniqueName: \"kubernetes.io/projected/1c367b83-3095-48d8-9a05-a481fc5bc516-kube-api-access-fbfcm\") pod \"maas-api-revoke-sub-e02c2f7d-4w7wq\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.485759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.485742    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02c2f7d-4w7wq\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.487694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.487666    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02c2f7d-4w7wq\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.510472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.510447    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbfcm\" (UniqueName: \"kubernetes.io/projected/1c367b83-3095-48d8-9a05-a481fc5bc516-kube-api-access-fbfcm\") pod \"maas-api-revoke-sub-e02c2f7d-4w7wq\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.615275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.615240    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:48.923829 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:48.923719    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c367b83_3095_48d8_9a05_a481fc5bc516.slice/crio-069da3c5a9adb6e5af3a1c11018ebb3cd027bd94fc7d6310266a2bf401397577 WatchSource:0}: Error finding container 069da3c5a9adb6e5af3a1c11018ebb3cd027bd94fc7d6310266a2bf401397577: Status 404 returned error can't find the container with id 069da3c5a9adb6e5af3a1c11018ebb3cd027bd94fc7d6310266a2bf401397577
Nov 02 14:26:48.926057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:48.926032    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"]
Nov 02 14:26:49.156189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.156155    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq" event={"ID":"1c367b83-3095-48d8-9a05-a481fc5bc516","Type":"ContainerStarted","Data":"069da3c5a9adb6e5af3a1c11018ebb3cd027bd94fc7d6310266a2bf401397577"}
Nov 02 14:26:49.443288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.443263    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:49.502201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.502165    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-openshift-ca\") pod \"296e26a3-3f71-4f75-89b9-4592599d94a3\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") "
Nov 02 14:26:49.502382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.502225    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-xks-ca\") pod \"296e26a3-3f71-4f75-89b9-4592599d94a3\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") "
Nov 02 14:26:49.502382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.502361    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/296e26a3-3f71-4f75-89b9-4592599d94a3-kube-api-access-vch6s\" (UniqueName: \"kubernetes.io/projected/296e26a3-3f71-4f75-89b9-4592599d94a3-kube-api-access-vch6s\") pod \"296e26a3-3f71-4f75-89b9-4592599d94a3\" (UID: \"296e26a3-3f71-4f75-89b9-4592599d94a3\") "
Nov 02 14:26:49.503453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.502643    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-openshift-ca" pod "296e26a3-3f71-4f75-89b9-4592599d94a3" (UID: "296e26a3-3f71-4f75-89b9-4592599d94a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:49.503453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.503010    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:49.506625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.505529    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-xks-ca" pod "296e26a3-3f71-4f75-89b9-4592599d94a3" (UID: "296e26a3-3f71-4f75-89b9-4592599d94a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:26:49.506625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.506125    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296e26a3-3f71-4f75-89b9-4592599d94a3-kube-api-access-vch6s" pod "296e26a3-3f71-4f75-89b9-4592599d94a3" (UID: "296e26a3-3f71-4f75-89b9-4592599d94a3"). InnerVolumeSpecName "kube-api-access-vch6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:49.605070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.604988    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/296e26a3-3f71-4f75-89b9-4592599d94a3-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:49.605070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:49.605016    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vch6s\" (UniqueName: \"kubernetes.io/projected/296e26a3-3f71-4f75-89b9-4592599d94a3-kube-api-access-vch6s\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:50.165577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:50.165502    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"
Nov 02 14:26:50.165577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:50.165537    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz" event={"ID":"296e26a3-3f71-4f75-89b9-4592599d94a3","Type":"ContainerDied","Data":"8465850f31589ff3d1459a6c917678bfe14b2b9c51778fd9f3f84f7803ebb744"}
Nov 02 14:26:50.166043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:50.165581    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8465850f31589ff3d1459a6c917678bfe14b2b9c51778fd9f3f84f7803ebb744"
Nov 02 14:26:50.167316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:50.167281    5034 generic.go:356] "Generic (PLEG): container finished" podID="1c367b83-3095-48d8-9a05-a481fc5bc516" containerID="600e0c0709cfd92adbf467f3c8dc867d24dfbddf106c06ad59f1004641bd6073" exitCode=0
Nov 02 14:26:50.167437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:50.167332    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq" event={"ID":"1c367b83-3095-48d8-9a05-a481fc5bc516","Type":"ContainerDied","Data":"600e0c0709cfd92adbf467f3c8dc867d24dfbddf106c06ad59f1004641bd6073"}
Nov 02 14:26:51.120608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.120571    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"]
Nov 02 14:26:51.377968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.377904    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:51.445562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.445517    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-xks-ca\") pod \"1c367b83-3095-48d8-9a05-a481fc5bc516\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") "
Nov 02 14:26:51.445769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.445624    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c367b83-3095-48d8-9a05-a481fc5bc516-kube-api-access-fbfcm\" (UniqueName: \"kubernetes.io/projected/1c367b83-3095-48d8-9a05-a481fc5bc516-kube-api-access-fbfcm\") pod \"1c367b83-3095-48d8-9a05-a481fc5bc516\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") "
Nov 02 14:26:51.445769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.445704    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-openshift-ca\") pod \"1c367b83-3095-48d8-9a05-a481fc5bc516\" (UID: \"1c367b83-3095-48d8-9a05-a481fc5bc516\") "
Nov 02 14:26:51.446443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.446410    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-openshift-ca" pod "1c367b83-3095-48d8-9a05-a481fc5bc516" (UID: "1c367b83-3095-48d8-9a05-a481fc5bc516"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:51.447106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.447063    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:51.448889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.448861    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c367b83-3095-48d8-9a05-a481fc5bc516-kube-api-access-fbfcm" pod "1c367b83-3095-48d8-9a05-a481fc5bc516" (UID: "1c367b83-3095-48d8-9a05-a481fc5bc516"). InnerVolumeSpecName "kube-api-access-fbfcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:51.451246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.451220    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-xks-ca" pod "1c367b83-3095-48d8-9a05-a481fc5bc516" (UID: "1c367b83-3095-48d8-9a05-a481fc5bc516"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:26:51.548357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.548321    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c367b83-3095-48d8-9a05-a481fc5bc516-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:51.548357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:51.548353    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbfcm\" (UniqueName: \"kubernetes.io/projected/1c367b83-3095-48d8-9a05-a481fc5bc516-kube-api-access-fbfcm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:52.188645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:52.188619    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"
Nov 02 14:26:52.188900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:52.188657    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq" event={"ID":"1c367b83-3095-48d8-9a05-a481fc5bc516","Type":"ContainerDied","Data":"069da3c5a9adb6e5af3a1c11018ebb3cd027bd94fc7d6310266a2bf401397577"}
Nov 02 14:26:52.188900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:52.188705    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="069da3c5a9adb6e5af3a1c11018ebb3cd027bd94fc7d6310266a2bf401397577"
Nov 02 14:26:56.121227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.121184    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b" podUID="8fbddffa-9066-4d71-ace7-39b307a80d37" containerName="payload-processing" containerID="cri-o://edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5" gracePeriod=25
Nov 02 14:26:56.417194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.417166    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:26:56.564485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.564449    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-plugins-config-volume\") pod \"8fbddffa-9066-4d71-ace7-39b307a80d37\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") "
Nov 02 14:26:56.564687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.564573    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-kube-api-access-xbw2v\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-kube-api-access-xbw2v\") pod \"8fbddffa-9066-4d71-ace7-39b307a80d37\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") "
Nov 02 14:26:56.564687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.564634    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-routing-overlay-volume\") pod \"8fbddffa-9066-4d71-ace7-39b307a80d37\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") "
Nov 02 14:26:56.564848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.564819    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-provider-credentials\") pod \"8fbddffa-9066-4d71-ace7-39b307a80d37\" (UID: \"8fbddffa-9066-4d71-ace7-39b307a80d37\") "
Nov 02 14:26:56.564963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.564942    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-plugins-config-volume" pod "8fbddffa-9066-4d71-ace7-39b307a80d37" (UID: "8fbddffa-9066-4d71-ace7-39b307a80d37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:56.565419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.565366    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-routing-overlay-volume" pod "8fbddffa-9066-4d71-ace7-39b307a80d37" (UID: "8fbddffa-9066-4d71-ace7-39b307a80d37"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:26:56.565926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.565641    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:56.565926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.565668    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8fbddffa-9066-4d71-ace7-39b307a80d37-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:56.567326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.567303    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-provider-credentials" pod "8fbddffa-9066-4d71-ace7-39b307a80d37" (UID: "8fbddffa-9066-4d71-ace7-39b307a80d37"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:56.567415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.567398    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-kube-api-access-xbw2v" pod "8fbddffa-9066-4d71-ace7-39b307a80d37" (UID: "8fbddffa-9066-4d71-ace7-39b307a80d37"). InnerVolumeSpecName "kube-api-access-xbw2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:26:56.667642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.667569    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbw2v\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-kube-api-access-xbw2v\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:56.667642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:56.667597    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8fbddffa-9066-4d71-ace7-39b307a80d37-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:26:57.245945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.245909    5034 generic.go:356] "Generic (PLEG): container finished" podID="8fbddffa-9066-4d71-ace7-39b307a80d37" containerID="edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5" exitCode=0
Nov 02 14:26:57.246406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.245958    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"
Nov 02 14:26:57.246406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.245963    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b" event={"ID":"8fbddffa-9066-4d71-ace7-39b307a80d37","Type":"ContainerDied","Data":"edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5"}
Nov 02 14:26:57.246406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.246089    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b" event={"ID":"8fbddffa-9066-4d71-ace7-39b307a80d37","Type":"ContainerDied","Data":"20144f8f3a6518fdf75dcfba6ccead9b94d05ef746bab9c28e1ec97427c5139c"}
Nov 02 14:26:57.246406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.246122    5034 scope.go:122] "RemoveContainer" containerID="edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5"
Nov 02 14:26:57.259330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.259307    5034 scope.go:122] "RemoveContainer" containerID="edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5"
Nov 02 14:26:57.259623 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:26:57.259600    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5\": container with ID starting with edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5 not found: ID does not exist" containerID="edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5"
Nov 02 14:26:57.259677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.259633    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5"} err="failed to get container status \"edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5\": rpc error: code = NotFound desc = could not find container \"edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5\": container with ID starting with edb0756fb6b6f6bd4729e8afb3c7e435b0a64fe94736393173055477de8dacd5 not found: ID does not exist"
Nov 02 14:26:57.306997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.306976    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"]
Nov 02 14:26:57.320890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.320868    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-965c9zm7b"]
Nov 02 14:26:57.359830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.359792    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"]
Nov 02 14:26:57.360652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.360636    5034 cpu_manager.go:414] "containerMap: removing container" podUID="296e26a3-3f71-4f75-89b9-4592599d94a3" containerName="revoke-keys"
Nov 02 14:26:57.360703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.360654    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="296e26a3-3f71-4f75-89b9-4592599d94a3" containerName="revoke-keys"
Nov 02 14:26:57.360703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.360666    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8fbddffa-9066-4d71-ace7-39b307a80d37" containerName="payload-processing"
Nov 02 14:26:57.360703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.360671    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbddffa-9066-4d71-ace7-39b307a80d37" containerName="payload-processing"
Nov 02 14:26:57.360703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.360686    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1c367b83-3095-48d8-9a05-a481fc5bc516" containerName="revoke-keys"
Nov 02 14:26:57.360703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.360692    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c367b83-3095-48d8-9a05-a481fc5bc516" containerName="revoke-keys"
Nov 02 14:26:57.360886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.360854    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="296e26a3-3f71-4f75-89b9-4592599d94a3" containerName="revoke-keys"
Nov 02 14:26:57.360886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.360864    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbddffa-9066-4d71-ace7-39b307a80d37" containerName="payload-processing"
Nov 02 14:26:57.360886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.360874    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c367b83-3095-48d8-9a05-a481fc5bc516" containerName="revoke-keys"
Nov 02 14:26:57.365897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.365878    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.382872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.382731    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:26:57.382872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.382831    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:26:57.382872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.382833    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:26:57.383105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.382900    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:26:57.385296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.385274    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"]
Nov 02 14:26:57.399809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.399774    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbddffa-9066-4d71-ace7-39b307a80d37" path="/var/lib/kubelet/pods/8fbddffa-9066-4d71-ace7-39b307a80d37/volumes"
Nov 02 14:26:57.485383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.485345    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0183dc5-ctl6n\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.485931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.485459    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l2rz\" (UniqueName: \"kubernetes.io/projected/10933ec6-c93c-42bd-aaf2-820dfd803ce1-kube-api-access-2l2rz\") pod \"maas-api-revoke-sub-b0183dc5-ctl6n\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.485931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.485494    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0183dc5-ctl6n\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.587224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.587139    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0183dc5-ctl6n\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.587393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.587249    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l2rz\" (UniqueName: \"kubernetes.io/projected/10933ec6-c93c-42bd-aaf2-820dfd803ce1-kube-api-access-2l2rz\") pod \"maas-api-revoke-sub-b0183dc5-ctl6n\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.587393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.587277    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0183dc5-ctl6n\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.587927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.587905    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0183dc5-ctl6n\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.589858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.589828    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0183dc5-ctl6n\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.612519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.612497    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l2rz\" (UniqueName: \"kubernetes.io/projected/10933ec6-c93c-42bd-aaf2-820dfd803ce1-kube-api-access-2l2rz\") pod \"maas-api-revoke-sub-b0183dc5-ctl6n\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.681209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.681187    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:26:57.994924 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:26:57.994894    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10933ec6_c93c_42bd_aaf2_820dfd803ce1.slice/crio-ff983328215f73006bcab9a41bcdb51b34ca5a6e5c8797a7a7d29b5f697111d4 WatchSource:0}: Error finding container ff983328215f73006bcab9a41bcdb51b34ca5a6e5c8797a7a7d29b5f697111d4: Status 404 returned error can't find the container with id ff983328215f73006bcab9a41bcdb51b34ca5a6e5c8797a7a7d29b5f697111d4
Nov 02 14:26:57.999947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:57.996847    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"]
Nov 02 14:26:58.255269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:58.255186    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n" event={"ID":"10933ec6-c93c-42bd-aaf2-820dfd803ce1","Type":"ContainerStarted","Data":"ff983328215f73006bcab9a41bcdb51b34ca5a6e5c8797a7a7d29b5f697111d4"}
Nov 02 14:26:59.268325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:59.268294    5034 generic.go:356] "Generic (PLEG): container finished" podID="10933ec6-c93c-42bd-aaf2-820dfd803ce1" containerID="ccf65bb76946ffeb8da05693861129aefd37e8d1475f65d19f660a878439a088" exitCode=0
Nov 02 14:26:59.268842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:26:59.268377    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n" event={"ID":"10933ec6-c93c-42bd-aaf2-820dfd803ce1","Type":"ContainerDied","Data":"ccf65bb76946ffeb8da05693861129aefd37e8d1475f65d19f660a878439a088"}
Nov 02 14:27:00.479808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.479748    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:27:00.536488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.536420    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-xks-ca\") pod \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") "
Nov 02 14:27:00.536641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.536510    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10933ec6-c93c-42bd-aaf2-820dfd803ce1-kube-api-access-2l2rz\" (UniqueName: \"kubernetes.io/projected/10933ec6-c93c-42bd-aaf2-820dfd803ce1-kube-api-access-2l2rz\") pod \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") "
Nov 02 14:27:00.536641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.536546    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-openshift-ca\") pod \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\" (UID: \"10933ec6-c93c-42bd-aaf2-820dfd803ce1\") "
Nov 02 14:27:00.537011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.536982    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-openshift-ca" pod "10933ec6-c93c-42bd-aaf2-820dfd803ce1" (UID: "10933ec6-c93c-42bd-aaf2-820dfd803ce1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:27:00.537344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.537326    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:00.539104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.539073    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-xks-ca" pod "10933ec6-c93c-42bd-aaf2-820dfd803ce1" (UID: "10933ec6-c93c-42bd-aaf2-820dfd803ce1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:27:00.539218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.539140    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10933ec6-c93c-42bd-aaf2-820dfd803ce1-kube-api-access-2l2rz" pod "10933ec6-c93c-42bd-aaf2-820dfd803ce1" (UID: "10933ec6-c93c-42bd-aaf2-820dfd803ce1"). InnerVolumeSpecName "kube-api-access-2l2rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:27:00.639119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.639082    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10933ec6-c93c-42bd-aaf2-820dfd803ce1-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:00.639119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.639109    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l2rz\" (UniqueName: \"kubernetes.io/projected/10933ec6-c93c-42bd-aaf2-820dfd803ce1-kube-api-access-2l2rz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:00.738347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.738307    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"]
Nov 02 14:27:00.739277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.739253    5034 cpu_manager.go:414] "containerMap: removing container" podUID="10933ec6-c93c-42bd-aaf2-820dfd803ce1" containerName="revoke-keys"
Nov 02 14:27:00.739277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.739275    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10933ec6-c93c-42bd-aaf2-820dfd803ce1" containerName="revoke-keys"
Nov 02 14:27:00.739542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.739527    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="10933ec6-c93c-42bd-aaf2-820dfd803ce1" containerName="revoke-keys"
Nov 02 14:27:00.743416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.743395    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:00.764972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.764947    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"]
Nov 02 14:27:00.842267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.842168    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3888b8c-mh647\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:00.842445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.842404    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3888b8c-mh647\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:00.842509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.842464    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/b799b069-64ec-49e5-a028-0dc5388f88cf-kube-api-access-pctff\") pod \"maas-api-revoke-sub-b3888b8c-mh647\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:00.944348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.944319    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3888b8c-mh647\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:00.944520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.944376    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/b799b069-64ec-49e5-a028-0dc5388f88cf-kube-api-access-pctff\") pod \"maas-api-revoke-sub-b3888b8c-mh647\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:00.944568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.944526    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3888b8c-mh647\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:00.945154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.945131    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3888b8c-mh647\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:00.946828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.946802    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3888b8c-mh647\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:00.970115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:00.970089    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/b799b069-64ec-49e5-a028-0dc5388f88cf-kube-api-access-pctff\") pod \"maas-api-revoke-sub-b3888b8c-mh647\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:01.058390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:01.058358    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:01.295865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:01.295817    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n" event={"ID":"10933ec6-c93c-42bd-aaf2-820dfd803ce1","Type":"ContainerDied","Data":"ff983328215f73006bcab9a41bcdb51b34ca5a6e5c8797a7a7d29b5f697111d4"}
Nov 02 14:27:01.296071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:01.295931    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff983328215f73006bcab9a41bcdb51b34ca5a6e5c8797a7a7d29b5f697111d4"
Nov 02 14:27:01.296071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:01.296022    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"
Nov 02 14:27:01.369615 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:27:01.369551    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb799b069_64ec_49e5_a028_0dc5388f88cf.slice/crio-1bf7352a74f406e62aeb7d194e57e31022fe417b39ebe2b7090d1d97d879d191 WatchSource:0}: Error finding container 1bf7352a74f406e62aeb7d194e57e31022fe417b39ebe2b7090d1d97d879d191: Status 404 returned error can't find the container with id 1bf7352a74f406e62aeb7d194e57e31022fe417b39ebe2b7090d1d97d879d191
Nov 02 14:27:01.371858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:01.371837    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"]
Nov 02 14:27:02.308593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:02.308547    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647" event={"ID":"b799b069-64ec-49e5-a028-0dc5388f88cf","Type":"ContainerStarted","Data":"2f05f7b153dcab8c6ef265acadfa2b3d75710466ebcce15bf4598b4a18520cf2"}
Nov 02 14:27:02.309122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:02.308618    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647" event={"ID":"b799b069-64ec-49e5-a028-0dc5388f88cf","Type":"ContainerStarted","Data":"1bf7352a74f406e62aeb7d194e57e31022fe417b39ebe2b7090d1d97d879d191"}
Nov 02 14:27:02.356164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:02.356115    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647" podStartSLOduration=2.356102324 podStartE2EDuration="2.356102324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:02.354956026 +0000 UTC m=+2613.106260464" watchObservedRunningTime="2026-11-02 14:27:02.356102324 +0000 UTC m=+2613.107406756"
Nov 02 14:27:03.321393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:03.321356    5034 generic.go:356] "Generic (PLEG): container finished" podID="b799b069-64ec-49e5-a028-0dc5388f88cf" containerID="2f05f7b153dcab8c6ef265acadfa2b3d75710466ebcce15bf4598b4a18520cf2" exitCode=0
Nov 02 14:27:03.321891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:03.321441    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647" event={"ID":"b799b069-64ec-49e5-a028-0dc5388f88cf","Type":"ContainerDied","Data":"2f05f7b153dcab8c6ef265acadfa2b3d75710466ebcce15bf4598b4a18520cf2"}
Nov 02 14:27:04.527109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.527079    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:04.599947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.599870    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b799b069-64ec-49e5-a028-0dc5388f88cf-kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/b799b069-64ec-49e5-a028-0dc5388f88cf-kube-api-access-pctff\") pod \"b799b069-64ec-49e5-a028-0dc5388f88cf\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") "
Nov 02 14:27:04.600250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.600228    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-openshift-ca\") pod \"b799b069-64ec-49e5-a028-0dc5388f88cf\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") "
Nov 02 14:27:04.600360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.600266    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-xks-ca\") pod \"b799b069-64ec-49e5-a028-0dc5388f88cf\" (UID: \"b799b069-64ec-49e5-a028-0dc5388f88cf\") "
Nov 02 14:27:04.600571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.600545    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-openshift-ca" pod "b799b069-64ec-49e5-a028-0dc5388f88cf" (UID: "b799b069-64ec-49e5-a028-0dc5388f88cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:27:04.601090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.601064    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:04.603332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.603272    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-xks-ca" pod "b799b069-64ec-49e5-a028-0dc5388f88cf" (UID: "b799b069-64ec-49e5-a028-0dc5388f88cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:27:04.603469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.603443    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b799b069-64ec-49e5-a028-0dc5388f88cf-kube-api-access-pctff" pod "b799b069-64ec-49e5-a028-0dc5388f88cf" (UID: "b799b069-64ec-49e5-a028-0dc5388f88cf"). InnerVolumeSpecName "kube-api-access-pctff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:27:04.703254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.703225    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b799b069-64ec-49e5-a028-0dc5388f88cf-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:04.703350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:04.703257    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pctff\" (UniqueName: \"kubernetes.io/projected/b799b069-64ec-49e5-a028-0dc5388f88cf-kube-api-access-pctff\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:05.341625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:05.341594    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"
Nov 02 14:27:05.341857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:05.341617    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647" event={"ID":"b799b069-64ec-49e5-a028-0dc5388f88cf","Type":"ContainerDied","Data":"1bf7352a74f406e62aeb7d194e57e31022fe417b39ebe2b7090d1d97d879d191"}
Nov 02 14:27:05.341857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:05.341674    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bf7352a74f406e62aeb7d194e57e31022fe417b39ebe2b7090d1d97d879d191"
Nov 02 14:27:18.428204 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:27:18.428167    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:27:25.810156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:25.810120    5034 generic.go:356] "Generic (PLEG): container finished" podID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerID="d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589" exitCode=0
Nov 02 14:27:25.810717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:25.810174    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" event={"ID":"55f5b619-f69a-45ff-904a-d8507d2c2ea6","Type":"ContainerDied","Data":"d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589"}
Nov 02 14:27:25.810717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:25.810664    5034 scope.go:122] "RemoveContainer" containerID="d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589"
Nov 02 14:27:25.812944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:25.812899    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:27:25.813077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:25.812982    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:27:25.813077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:25.813031    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:27:26.821485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:26.821450    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" event={"ID":"55f5b619-f69a-45ff-904a-d8507d2c2ea6","Type":"ContainerStarted","Data":"71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011"}
Nov 02 14:27:27.159611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:27.159523    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:27:27.161102 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:27.161073    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 14:27:27.161265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:27.161131    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 14:27:27.850447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:27.850416    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:27:28.159983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:28.159900    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 14:27:28.159983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:28.159959    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 14:27:29.200815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:29.190892    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 14:27:29.200815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:29.190966    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 14:27:30.159939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:30.159905    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 14:27:30.160149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:30.159961    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 14:27:31.160378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:31.160344    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 14:27:31.160802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:31.160403    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 14:27:32.160189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:32.160152    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused" start-of-body=
Nov 02 14:27:32.160380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:32.160216    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.28:15021/healthz/ready\": dial tcp 10.132.1.28:15021: connect: connection refused"
Nov 02 14:27:33.164022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:33.163985    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:27:33.165010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:33.164990    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:27:36.943810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:36.943299    5034 scope.go:122] "RemoveContainer" containerID="34b024d5dfa0c4631c7049a8841c86a0c63cd45cdf862fd2ddeb314a3a7980d3"
Nov 02 14:27:37.027814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:37.024970    5034 scope.go:122] "RemoveContainer" containerID="de158d18fbb6251b37826e6a57e3c1ec9ca8cd4ccd73716886187cead5b0a894"
Nov 02 14:27:42.698973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.698891    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"]
Nov 02 14:27:42.699951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.699931    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b799b069-64ec-49e5-a028-0dc5388f88cf" containerName="revoke-keys"
Nov 02 14:27:42.699951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.699952    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b799b069-64ec-49e5-a028-0dc5388f88cf" containerName="revoke-keys"
Nov 02 14:27:42.700099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.700090    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b799b069-64ec-49e5-a028-0dc5388f88cf" containerName="revoke-keys"
Nov 02 14:27:42.705758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.705739    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:42.720073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.720044    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"]
Nov 02 14:27:42.722417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.722391    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:27:42.727721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.727693    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:27:42.728181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.728160    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:27:42.728360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.728340    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:27:42.886674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.886637    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80b02f5-nq6g8\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:42.886674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.886674    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4tw7\" (UniqueName: \"kubernetes.io/projected/1a4de6d4-6060-4cee-b97f-6537e38a7592-kube-api-access-d4tw7\") pod \"maas-api-revoke-sub-e80b02f5-nq6g8\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:42.887005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.886917    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80b02f5-nq6g8\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:42.988257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.988230    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80b02f5-nq6g8\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:42.988432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.988262    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4tw7\" (UniqueName: \"kubernetes.io/projected/1a4de6d4-6060-4cee-b97f-6537e38a7592-kube-api-access-d4tw7\") pod \"maas-api-revoke-sub-e80b02f5-nq6g8\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:42.988506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.988437    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80b02f5-nq6g8\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:42.988995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.988973    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e80b02f5-nq6g8\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:42.991082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:42.991062    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e80b02f5-nq6g8\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:43.014338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:43.014313    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4tw7\" (UniqueName: \"kubernetes.io/projected/1a4de6d4-6060-4cee-b97f-6537e38a7592-kube-api-access-d4tw7\") pod \"maas-api-revoke-sub-e80b02f5-nq6g8\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:43.022943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:43.022924    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:43.337168 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:27:43.337136    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4de6d4_6060_4cee_b97f_6537e38a7592.slice/crio-8ccc4d5310212dfe99f246e9d8b897d1be7666f941e07ede1ec02e1adcc9c6ee WatchSource:0}: Error finding container 8ccc4d5310212dfe99f246e9d8b897d1be7666f941e07ede1ec02e1adcc9c6ee: Status 404 returned error can't find the container with id 8ccc4d5310212dfe99f246e9d8b897d1be7666f941e07ede1ec02e1adcc9c6ee
Nov 02 14:27:43.340154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:43.340126    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"]
Nov 02 14:27:44.127396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:44.127364    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8" event={"ID":"1a4de6d4-6060-4cee-b97f-6537e38a7592","Type":"ContainerStarted","Data":"c0dedafc2900c2e4ea8121f96f0d8f579ce5c21eae5d15e11a150ffefb20825e"}
Nov 02 14:27:44.127764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:44.127415    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8" event={"ID":"1a4de6d4-6060-4cee-b97f-6537e38a7592","Type":"ContainerStarted","Data":"8ccc4d5310212dfe99f246e9d8b897d1be7666f941e07ede1ec02e1adcc9c6ee"}
Nov 02 14:27:44.177906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:44.177852    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8" podStartSLOduration=2.177838751 podStartE2EDuration="2.177838751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:44.177156973 +0000 UTC m=+2654.928461411" watchObservedRunningTime="2026-11-02 14:27:44.177838751 +0000 UTC m=+2654.929143190"
Nov 02 14:27:45.148667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:45.148620    5034 generic.go:356] "Generic (PLEG): container finished" podID="1a4de6d4-6060-4cee-b97f-6537e38a7592" containerID="c0dedafc2900c2e4ea8121f96f0d8f579ce5c21eae5d15e11a150ffefb20825e" exitCode=0
Nov 02 14:27:45.149145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:45.148687    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8" event={"ID":"1a4de6d4-6060-4cee-b97f-6537e38a7592","Type":"ContainerDied","Data":"c0dedafc2900c2e4ea8121f96f0d8f579ce5c21eae5d15e11a150ffefb20825e"}
Nov 02 14:27:46.343264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.343234    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:46.441856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.441764    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a4de6d4-6060-4cee-b97f-6537e38a7592-kube-api-access-d4tw7\" (UniqueName: \"kubernetes.io/projected/1a4de6d4-6060-4cee-b97f-6537e38a7592-kube-api-access-d4tw7\") pod \"1a4de6d4-6060-4cee-b97f-6537e38a7592\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") "
Nov 02 14:27:46.442015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.441909    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-xks-ca\") pod \"1a4de6d4-6060-4cee-b97f-6537e38a7592\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") "
Nov 02 14:27:46.442015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.441963    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-openshift-ca\") pod \"1a4de6d4-6060-4cee-b97f-6537e38a7592\" (UID: \"1a4de6d4-6060-4cee-b97f-6537e38a7592\") "
Nov 02 14:27:46.442613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.442569    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-openshift-ca" pod "1a4de6d4-6060-4cee-b97f-6537e38a7592" (UID: "1a4de6d4-6060-4cee-b97f-6537e38a7592"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:27:46.443936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.443856    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.445387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.445317    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4de6d4-6060-4cee-b97f-6537e38a7592-kube-api-access-d4tw7" pod "1a4de6d4-6060-4cee-b97f-6537e38a7592" (UID: "1a4de6d4-6060-4cee-b97f-6537e38a7592"). InnerVolumeSpecName "kube-api-access-d4tw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:27:46.445547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.445526    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-xks-ca" pod "1a4de6d4-6060-4cee-b97f-6537e38a7592" (UID: "1a4de6d4-6060-4cee-b97f-6537e38a7592"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:27:46.545354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.545325    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4tw7\" (UniqueName: \"kubernetes.io/projected/1a4de6d4-6060-4cee-b97f-6537e38a7592-kube-api-access-d4tw7\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:46.545354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:46.545351    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a4de6d4-6060-4cee-b97f-6537e38a7592-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:47.166230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:47.166201    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"
Nov 02 14:27:47.166413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:47.166226    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8" event={"ID":"1a4de6d4-6060-4cee-b97f-6537e38a7592","Type":"ContainerDied","Data":"8ccc4d5310212dfe99f246e9d8b897d1be7666f941e07ede1ec02e1adcc9c6ee"}
Nov 02 14:27:47.166413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:47.166274    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ccc4d5310212dfe99f246e9d8b897d1be7666f941e07ede1ec02e1adcc9c6ee"
Nov 02 14:27:48.417638 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:27:48.417603    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:27:55.594756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.594724    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"]
Nov 02 14:27:55.595842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.595817    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1a4de6d4-6060-4cee-b97f-6537e38a7592" containerName="revoke-keys"
Nov 02 14:27:55.595842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.595841    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a4de6d4-6060-4cee-b97f-6537e38a7592" containerName="revoke-keys"
Nov 02 14:27:55.596106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.596090    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a4de6d4-6060-4cee-b97f-6537e38a7592" containerName="revoke-keys"
Nov 02 14:27:55.600209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.600187    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:55.628317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.628297    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:27:55.635974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.635949    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:27:55.635974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.635960    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:27:55.636168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.636003    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:27:55.641486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.641464    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"]
Nov 02 14:27:55.779336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.779299    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afcb5276-cr28w\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:55.779503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.779475    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afcb5276-cr28w\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:55.779598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.779547    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2tc\" (UniqueName: \"kubernetes.io/projected/6b6a675a-9567-48bf-aa35-07099224550a-kube-api-access-bb2tc\") pod \"maas-api-revoke-sub-afcb5276-cr28w\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:55.899710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.899623    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afcb5276-cr28w\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:55.900071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.900044    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afcb5276-cr28w\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:55.915709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.915680    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2tc\" (UniqueName: \"kubernetes.io/projected/6b6a675a-9567-48bf-aa35-07099224550a-kube-api-access-bb2tc\") pod \"maas-api-revoke-sub-afcb5276-cr28w\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:55.916020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.915924    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afcb5276-cr28w\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:55.945267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.945198    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afcb5276-cr28w\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:55.965490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:55.965423    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2tc\" (UniqueName: \"kubernetes.io/projected/6b6a675a-9567-48bf-aa35-07099224550a-kube-api-access-bb2tc\") pod \"maas-api-revoke-sub-afcb5276-cr28w\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:56.217353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:56.217276    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:56.532533 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:27:56.532506    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6a675a_9567_48bf_aa35_07099224550a.slice/crio-99f01f1c1b601b0531112833614bfbbf10ba48437d677281424573ec75414d78 WatchSource:0}: Error finding container 99f01f1c1b601b0531112833614bfbbf10ba48437d677281424573ec75414d78: Status 404 returned error can't find the container with id 99f01f1c1b601b0531112833614bfbbf10ba48437d677281424573ec75414d78
Nov 02 14:27:56.533687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:56.533654    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"]
Nov 02 14:27:57.352907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:57.352871    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w" event={"ID":"6b6a675a-9567-48bf-aa35-07099224550a","Type":"ContainerStarted","Data":"3e4387897a21534bd4a5c53ee4ba9a5141f42b3bf14e278d519d0ada97ab1d13"}
Nov 02 14:27:57.354097 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:57.354066    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w" event={"ID":"6b6a675a-9567-48bf-aa35-07099224550a","Type":"ContainerStarted","Data":"99f01f1c1b601b0531112833614bfbbf10ba48437d677281424573ec75414d78"}
Nov 02 14:27:57.402514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:57.402458    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w" podStartSLOduration=2.402444478 podStartE2EDuration="2.402444478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:27:57.39809217 +0000 UTC m=+2668.149396608" watchObservedRunningTime="2026-11-02 14:27:57.402444478 +0000 UTC m=+2668.153748916"
Nov 02 14:27:58.362558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:58.362523    5034 generic.go:356] "Generic (PLEG): container finished" podID="6b6a675a-9567-48bf-aa35-07099224550a" containerID="3e4387897a21534bd4a5c53ee4ba9a5141f42b3bf14e278d519d0ada97ab1d13" exitCode=0
Nov 02 14:27:58.362988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:58.362613    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w" event={"ID":"6b6a675a-9567-48bf-aa35-07099224550a","Type":"ContainerDied","Data":"3e4387897a21534bd4a5c53ee4ba9a5141f42b3bf14e278d519d0ada97ab1d13"}
Nov 02 14:27:59.554176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.554147    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:27:59.679308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.679220    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-openshift-ca\") pod \"6b6a675a-9567-48bf-aa35-07099224550a\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") "
Nov 02 14:27:59.679508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.679417    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6a675a-9567-48bf-aa35-07099224550a-kube-api-access-bb2tc\" (UniqueName: \"kubernetes.io/projected/6b6a675a-9567-48bf-aa35-07099224550a-kube-api-access-bb2tc\") pod \"6b6a675a-9567-48bf-aa35-07099224550a\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") "
Nov 02 14:27:59.679575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.679516    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-xks-ca\") pod \"6b6a675a-9567-48bf-aa35-07099224550a\" (UID: \"6b6a675a-9567-48bf-aa35-07099224550a\") "
Nov 02 14:27:59.679672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.679644    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-openshift-ca" pod "6b6a675a-9567-48bf-aa35-07099224550a" (UID: "6b6a675a-9567-48bf-aa35-07099224550a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:27:59.680464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.680440    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:59.682046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.682018    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6a675a-9567-48bf-aa35-07099224550a-kube-api-access-bb2tc" pod "6b6a675a-9567-48bf-aa35-07099224550a" (UID: "6b6a675a-9567-48bf-aa35-07099224550a"). InnerVolumeSpecName "kube-api-access-bb2tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:27:59.682256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.682230    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-xks-ca" pod "6b6a675a-9567-48bf-aa35-07099224550a" (UID: "6b6a675a-9567-48bf-aa35-07099224550a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:27:59.782076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.782044    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb2tc\" (UniqueName: \"kubernetes.io/projected/6b6a675a-9567-48bf-aa35-07099224550a-kube-api-access-bb2tc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:27:59.782076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:27:59.782073    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6a675a-9567-48bf-aa35-07099224550a-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:00.383910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:00.383875    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w" event={"ID":"6b6a675a-9567-48bf-aa35-07099224550a","Type":"ContainerDied","Data":"99f01f1c1b601b0531112833614bfbbf10ba48437d677281424573ec75414d78"}
Nov 02 14:28:00.383910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:00.383895    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"
Nov 02 14:28:00.384133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:00.383930    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f01f1c1b601b0531112833614bfbbf10ba48437d677281424573ec75414d78"
Nov 02 14:28:07.582822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.571088    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"]
Nov 02 14:28:07.582822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.572518    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6b6a675a-9567-48bf-aa35-07099224550a" containerName="revoke-keys"
Nov 02 14:28:07.582822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.572540    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6a675a-9567-48bf-aa35-07099224550a" containerName="revoke-keys"
Nov 02 14:28:07.582822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.572758    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6a675a-9567-48bf-aa35-07099224550a" containerName="revoke-keys"
Nov 02 14:28:07.604858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.597308    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:07.605339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.605312    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"]
Nov 02 14:28:07.620191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.620164    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:28:07.640678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.638927    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:28:07.640678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.639059    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:28:07.640678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.639125    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:28:07.686230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.686187    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mpwj\" (UniqueName: \"kubernetes.io/projected/76da187f-6f50-4923-8760-a158d597e841-kube-api-access-9mpwj\") pod \"maas-api-revoke-sub-d0e4d71d-tv74n\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:07.686944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.686622    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76da187f-6f50-4923-8760-a158d597e841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e4d71d-tv74n\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:07.687386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.687309    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76da187f-6f50-4923-8760-a158d597e841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e4d71d-tv74n\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:07.789047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.789015    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76da187f-6f50-4923-8760-a158d597e841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e4d71d-tv74n\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:07.789302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.789275    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mpwj\" (UniqueName: \"kubernetes.io/projected/76da187f-6f50-4923-8760-a158d597e841-kube-api-access-9mpwj\") pod \"maas-api-revoke-sub-d0e4d71d-tv74n\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:07.789470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.789353    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76da187f-6f50-4923-8760-a158d597e841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e4d71d-tv74n\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:07.789863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.789835    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76da187f-6f50-4923-8760-a158d597e841-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e4d71d-tv74n\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:07.792196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.792171    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76da187f-6f50-4923-8760-a158d597e841-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e4d71d-tv74n\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:07.814877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:07.814846    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mpwj\" (UniqueName: \"kubernetes.io/projected/76da187f-6f50-4923-8760-a158d597e841-kube-api-access-9mpwj\") pod \"maas-api-revoke-sub-d0e4d71d-tv74n\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:08.039930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:08.039899    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:08.095806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:08.090729    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"]
Nov 02 14:28:08.379848 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:28:08.379772    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76da187f_6f50_4923_8760_a158d597e841.slice/crio-c232a305d20ac3297b5f9148b54217678147c658565887c150562a5e932a1b18 WatchSource:0}: Error finding container c232a305d20ac3297b5f9148b54217678147c658565887c150562a5e932a1b18: Status 404 returned error can't find the container with id c232a305d20ac3297b5f9148b54217678147c658565887c150562a5e932a1b18
Nov 02 14:28:08.383457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:08.383429    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"]
Nov 02 14:28:08.470839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:08.470810    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n" event={"ID":"76da187f-6f50-4923-8760-a158d597e841","Type":"ContainerStarted","Data":"c232a305d20ac3297b5f9148b54217678147c658565887c150562a5e932a1b18"}
Nov 02 14:28:09.487746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:09.487710    5034 generic.go:356] "Generic (PLEG): container finished" podID="76da187f-6f50-4923-8760-a158d597e841" containerID="ee105bae23b96aa18b54a22abfc34a99e3d03bf6d22f72a3006a10bfdb81c481" exitCode=0
Nov 02 14:28:09.488227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:09.487812    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n" event={"ID":"76da187f-6f50-4923-8760-a158d597e841","Type":"ContainerDied","Data":"ee105bae23b96aa18b54a22abfc34a99e3d03bf6d22f72a3006a10bfdb81c481"}
Nov 02 14:28:10.754750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:10.754682    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:10.856511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:10.856465    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76da187f-6f50-4923-8760-a158d597e841-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76da187f-6f50-4923-8760-a158d597e841-maas-api-openshift-ca\") pod \"76da187f-6f50-4923-8760-a158d597e841\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") "
Nov 02 14:28:10.856851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:10.856828    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76da187f-6f50-4923-8760-a158d597e841-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76da187f-6f50-4923-8760-a158d597e841-maas-api-xks-ca\") pod \"76da187f-6f50-4923-8760-a158d597e841\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") "
Nov 02 14:28:10.880800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:10.857107    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76da187f-6f50-4923-8760-a158d597e841-kube-api-access-9mpwj\" (UniqueName: \"kubernetes.io/projected/76da187f-6f50-4923-8760-a158d597e841-kube-api-access-9mpwj\") pod \"76da187f-6f50-4923-8760-a158d597e841\" (UID: \"76da187f-6f50-4923-8760-a158d597e841\") "
Nov 02 14:28:10.883064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:10.867283    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76da187f-6f50-4923-8760-a158d597e841-maas-api-openshift-ca" pod "76da187f-6f50-4923-8760-a158d597e841" (UID: "76da187f-6f50-4923-8760-a158d597e841"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:10.922812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:10.922649    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76da187f-6f50-4923-8760-a158d597e841-kube-api-access-9mpwj" pod "76da187f-6f50-4923-8760-a158d597e841" (UID: "76da187f-6f50-4923-8760-a158d597e841"). InnerVolumeSpecName "kube-api-access-9mpwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:10.948014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:10.934213    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76da187f-6f50-4923-8760-a158d597e841-maas-api-xks-ca" pod "76da187f-6f50-4923-8760-a158d597e841" (UID: "76da187f-6f50-4923-8760-a158d597e841"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:28:11.076837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:11.076663    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mpwj\" (UniqueName: \"kubernetes.io/projected/76da187f-6f50-4923-8760-a158d597e841-kube-api-access-9mpwj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:11.076837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:11.076696    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76da187f-6f50-4923-8760-a158d597e841-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:11.076837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:11.076711    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76da187f-6f50-4923-8760-a158d597e841-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:11.569208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:11.568891    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n" event={"ID":"76da187f-6f50-4923-8760-a158d597e841","Type":"ContainerDied","Data":"c232a305d20ac3297b5f9148b54217678147c658565887c150562a5e932a1b18"}
Nov 02 14:28:11.569208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:11.568956    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c232a305d20ac3297b5f9148b54217678147c658565887c150562a5e932a1b18"
Nov 02 14:28:11.569208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:11.569024    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"
Nov 02 14:28:13.097856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.097818    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s" podUID="b32758a5-836f-492a-aa2e-26653d3ca2a2" containerName="payload-processing" containerID="cri-o://511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2" gracePeriod=25
Nov 02 14:28:13.414752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.414724    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:28:13.426456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.426429    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-routing-overlay-volume\") pod \"b32758a5-836f-492a-aa2e-26653d3ca2a2\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") "
Nov 02 14:28:13.426575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.426508    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-provider-credentials\") pod \"b32758a5-836f-492a-aa2e-26653d3ca2a2\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") "
Nov 02 14:28:13.426671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.426657    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-kube-api-access-kfhnp\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-kube-api-access-kfhnp\") pod \"b32758a5-836f-492a-aa2e-26653d3ca2a2\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") "
Nov 02 14:28:13.426765 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.426751    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-plugins-config-volume\") pod \"b32758a5-836f-492a-aa2e-26653d3ca2a2\" (UID: \"b32758a5-836f-492a-aa2e-26653d3ca2a2\") "
Nov 02 14:28:13.426847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.426815    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-routing-overlay-volume" pod "b32758a5-836f-492a-aa2e-26653d3ca2a2" (UID: "b32758a5-836f-492a-aa2e-26653d3ca2a2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:13.427176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.427152    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-plugins-config-volume" pod "b32758a5-836f-492a-aa2e-26653d3ca2a2" (UID: "b32758a5-836f-492a-aa2e-26653d3ca2a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:13.427876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.427855    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:13.427948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.427883    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b32758a5-836f-492a-aa2e-26653d3ca2a2-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:13.428773 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.428750    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-kube-api-access-kfhnp" pod "b32758a5-836f-492a-aa2e-26653d3ca2a2" (UID: "b32758a5-836f-492a-aa2e-26653d3ca2a2"). InnerVolumeSpecName "kube-api-access-kfhnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:13.428994 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.428843    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-provider-credentials" pod "b32758a5-836f-492a-aa2e-26653d3ca2a2" (UID: "b32758a5-836f-492a-aa2e-26653d3ca2a2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:13.529181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.529149    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:13.529181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.529174    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfhnp\" (UniqueName: \"kubernetes.io/projected/b32758a5-836f-492a-aa2e-26653d3ca2a2-kube-api-access-kfhnp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:13.589458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.589422    5034 generic.go:356] "Generic (PLEG): container finished" podID="b32758a5-836f-492a-aa2e-26653d3ca2a2" containerID="511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2" exitCode=0
Nov 02 14:28:13.589640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.589481    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"
Nov 02 14:28:13.589640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.589503    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s" event={"ID":"b32758a5-836f-492a-aa2e-26653d3ca2a2","Type":"ContainerDied","Data":"511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2"}
Nov 02 14:28:13.589640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.589564    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s" event={"ID":"b32758a5-836f-492a-aa2e-26653d3ca2a2","Type":"ContainerDied","Data":"a3245cc1f7bbdebf95b0858084f3b2f0fbf8324eac98f22ecf3775b0681eb7cd"}
Nov 02 14:28:13.589640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.589590    5034 scope.go:122] "RemoveContainer" containerID="511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2"
Nov 02 14:28:13.604618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.604592    5034 scope.go:122] "RemoveContainer" containerID="511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2"
Nov 02 14:28:13.604907 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:28:13.604883    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2\": container with ID starting with 511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2 not found: ID does not exist" containerID="511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2"
Nov 02 14:28:13.604985 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.604914    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2"} err="failed to get container status \"511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2\": rpc error: code = NotFound desc = could not find container \"511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2\": container with ID starting with 511db77916e5e8e6da930a369aa88fb08c68552e3ea9bbee7f6f74d27ca4c1c2 not found: ID does not exist"
Nov 02 14:28:13.667840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.667731    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"]
Nov 02 14:28:13.685710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:13.685462    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2d13f4/payload-processing-external-model-e2e-extmod-a-2d13f4-5964tgm8s"]
Nov 02 14:28:15.440208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:15.440162    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32758a5-836f-492a-aa2e-26653d3ca2a2" path="/var/lib/kubelet/pods/b32758a5-836f-492a-aa2e-26653d3ca2a2/volumes"
Nov 02 14:28:18.415510 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:28:18.415474    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:28:18.978351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:18.978313    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"]
Nov 02 14:28:23.978986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:23.978948    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w" podUID="8007ba35-5dca-48bc-8401-62896c8af6e8" containerName="payload-processing" containerID="cri-o://eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346" gracePeriod=25
Nov 02 14:28:24.279564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.279536    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:28:24.438757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.438724    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-plugins-config-volume\") pod \"8007ba35-5dca-48bc-8401-62896c8af6e8\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") "
Nov 02 14:28:24.438927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.438913    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-provider-credentials\") pod \"8007ba35-5dca-48bc-8401-62896c8af6e8\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") "
Nov 02 14:28:24.439002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.438942    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-kube-api-access-f2vsk\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-kube-api-access-f2vsk\") pod \"8007ba35-5dca-48bc-8401-62896c8af6e8\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") "
Nov 02 14:28:24.439002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.438978    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-routing-overlay-volume\") pod \"8007ba35-5dca-48bc-8401-62896c8af6e8\" (UID: \"8007ba35-5dca-48bc-8401-62896c8af6e8\") "
Nov 02 14:28:24.439200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.439171    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-plugins-config-volume" pod "8007ba35-5dca-48bc-8401-62896c8af6e8" (UID: "8007ba35-5dca-48bc-8401-62896c8af6e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:24.439397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.439373    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-routing-overlay-volume" pod "8007ba35-5dca-48bc-8401-62896c8af6e8" (UID: "8007ba35-5dca-48bc-8401-62896c8af6e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:28:24.440283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.440255    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:24.440433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.440286    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8007ba35-5dca-48bc-8401-62896c8af6e8-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:24.441330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.441303    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-provider-credentials" pod "8007ba35-5dca-48bc-8401-62896c8af6e8" (UID: "8007ba35-5dca-48bc-8401-62896c8af6e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:24.441428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.441367    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-kube-api-access-f2vsk" pod "8007ba35-5dca-48bc-8401-62896c8af6e8" (UID: "8007ba35-5dca-48bc-8401-62896c8af6e8"). InnerVolumeSpecName "kube-api-access-f2vsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:28:24.541490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.541412    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:24.541490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.541442    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2vsk\" (UniqueName: \"kubernetes.io/projected/8007ba35-5dca-48bc-8401-62896c8af6e8-kube-api-access-f2vsk\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:28:24.699748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.699712    5034 generic.go:356] "Generic (PLEG): container finished" podID="8007ba35-5dca-48bc-8401-62896c8af6e8" containerID="eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346" exitCode=0
Nov 02 14:28:24.699938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.699767    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"
Nov 02 14:28:24.699938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.699811    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w" event={"ID":"8007ba35-5dca-48bc-8401-62896c8af6e8","Type":"ContainerDied","Data":"eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346"}
Nov 02 14:28:24.699938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.699886    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w" event={"ID":"8007ba35-5dca-48bc-8401-62896c8af6e8","Type":"ContainerDied","Data":"79acc9ff5eb3053f22f542fef05f704f983baf49f44eb7bb8b1cce14bd9f7197"}
Nov 02 14:28:24.699938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.699914    5034 scope.go:122] "RemoveContainer" containerID="eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346"
Nov 02 14:28:24.712793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.712751    5034 scope.go:122] "RemoveContainer" containerID="eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346"
Nov 02 14:28:24.713113 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:28:24.713088    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346\": container with ID starting with eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346 not found: ID does not exist" containerID="eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346"
Nov 02 14:28:24.713198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.713124    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346"} err="failed to get container status \"eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346\": rpc error: code = NotFound desc = could not find container \"eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346\": container with ID starting with eaa04c9557d95e0bc7a758929bc52d2c6d7116d1b55ee26c50983494146ca346 not found: ID does not exist"
Nov 02 14:28:24.759180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.759156    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"]
Nov 02 14:28:24.763738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:24.763719    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-866drfm2w"]
Nov 02 14:28:25.398229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:25.398196    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8007ba35-5dca-48bc-8401-62896c8af6e8" path="/var/lib/kubelet/pods/8007ba35-5dca-48bc-8401-62896c8af6e8/volumes"
Nov 02 14:28:30.583083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:30.583048    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:28:30.586057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:28:30.586030    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:28:48.416211 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:28:48.416171    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:29:02.217076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.217043    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"]
Nov 02 14:29:02.217940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.217918    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8007ba35-5dca-48bc-8401-62896c8af6e8" containerName="payload-processing"
Nov 02 14:29:02.217940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.217935    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8007ba35-5dca-48bc-8401-62896c8af6e8" containerName="payload-processing"
Nov 02 14:29:02.218124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.217957    5034 cpu_manager.go:414] "containerMap: removing container" podUID="76da187f-6f50-4923-8760-a158d597e841" containerName="revoke-keys"
Nov 02 14:29:02.218124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.217962    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76da187f-6f50-4923-8760-a158d597e841" containerName="revoke-keys"
Nov 02 14:29:02.218124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.217977    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b32758a5-836f-492a-aa2e-26653d3ca2a2" containerName="payload-processing"
Nov 02 14:29:02.218124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.217984    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32758a5-836f-492a-aa2e-26653d3ca2a2" containerName="payload-processing"
Nov 02 14:29:02.218265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.218133    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8007ba35-5dca-48bc-8401-62896c8af6e8" containerName="payload-processing"
Nov 02 14:29:02.218265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.218152    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="76da187f-6f50-4923-8760-a158d597e841" containerName="revoke-keys"
Nov 02 14:29:02.218265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.218160    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32758a5-836f-492a-aa2e-26653d3ca2a2" containerName="payload-processing"
Nov 02 14:29:02.222017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.222000    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.242664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.242635    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"]
Nov 02 14:29:02.251828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.249864    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:29:02.251828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.249874    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:29:02.307721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.307679    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:29:02.308003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.307803    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:29:02.387458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.387426    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5abfc98-64xbh\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.387772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.387751    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5abfc98-64xbh\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.387879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.387849    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hk9\" (UniqueName: \"kubernetes.io/projected/774ec3a9-cda4-4ea3-a028-c23476678fdb-kube-api-access-f6hk9\") pod \"maas-api-revoke-sub-c5abfc98-64xbh\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.496286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.496246    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5abfc98-64xbh\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.496564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.496538    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hk9\" (UniqueName: \"kubernetes.io/projected/774ec3a9-cda4-4ea3-a028-c23476678fdb-kube-api-access-f6hk9\") pod \"maas-api-revoke-sub-c5abfc98-64xbh\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.496929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.496905    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5abfc98-64xbh\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.498076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.498047    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5abfc98-64xbh\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.500334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.500306    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5abfc98-64xbh\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.529070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.529047    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hk9\" (UniqueName: \"kubernetes.io/projected/774ec3a9-cda4-4ea3-a028-c23476678fdb-kube-api-access-f6hk9\") pod \"maas-api-revoke-sub-c5abfc98-64xbh\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:02.536933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:02.536911    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:03.018142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:03.016957    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"]
Nov 02 14:29:03.446692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:03.446644    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh" event={"ID":"774ec3a9-cda4-4ea3-a028-c23476678fdb","Type":"ContainerStarted","Data":"f2c65b63fcf51a63c111b8a33d11fccc873014c2b3d6d092158e59a2e3b344b2"}
Nov 02 14:29:04.456637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:04.456562    5034 generic.go:356] "Generic (PLEG): container finished" podID="774ec3a9-cda4-4ea3-a028-c23476678fdb" containerID="89d9db0b0df685b258dadb2fb70811a4c93c46818cc0a74980768f369712d252" exitCode=0
Nov 02 14:29:04.457078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:04.456645    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh" event={"ID":"774ec3a9-cda4-4ea3-a028-c23476678fdb","Type":"ContainerDied","Data":"89d9db0b0df685b258dadb2fb70811a4c93c46818cc0a74980768f369712d252"}
Nov 02 14:29:05.636980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.636950    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:05.653820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.653774    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-xks-ca\") pod \"774ec3a9-cda4-4ea3-a028-c23476678fdb\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") "
Nov 02 14:29:05.654052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.654033    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/774ec3a9-cda4-4ea3-a028-c23476678fdb-kube-api-access-f6hk9\" (UniqueName: \"kubernetes.io/projected/774ec3a9-cda4-4ea3-a028-c23476678fdb-kube-api-access-f6hk9\") pod \"774ec3a9-cda4-4ea3-a028-c23476678fdb\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") "
Nov 02 14:29:05.654123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.654103    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-openshift-ca\") pod \"774ec3a9-cda4-4ea3-a028-c23476678fdb\" (UID: \"774ec3a9-cda4-4ea3-a028-c23476678fdb\") "
Nov 02 14:29:05.654469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.654446    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-openshift-ca" pod "774ec3a9-cda4-4ea3-a028-c23476678fdb" (UID: "774ec3a9-cda4-4ea3-a028-c23476678fdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:05.654640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.654623    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:05.656244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.656216    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-xks-ca" pod "774ec3a9-cda4-4ea3-a028-c23476678fdb" (UID: "774ec3a9-cda4-4ea3-a028-c23476678fdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:29:05.656482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.656456    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774ec3a9-cda4-4ea3-a028-c23476678fdb-kube-api-access-f6hk9" pod "774ec3a9-cda4-4ea3-a028-c23476678fdb" (UID: "774ec3a9-cda4-4ea3-a028-c23476678fdb"). InnerVolumeSpecName "kube-api-access-f6hk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:05.755615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.755588    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hk9\" (UniqueName: \"kubernetes.io/projected/774ec3a9-cda4-4ea3-a028-c23476678fdb-kube-api-access-f6hk9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:05.755615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:05.755615    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/774ec3a9-cda4-4ea3-a028-c23476678fdb-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:06.473884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:06.473853    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"
Nov 02 14:29:06.474089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:06.473893    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh" event={"ID":"774ec3a9-cda4-4ea3-a028-c23476678fdb","Type":"ContainerDied","Data":"f2c65b63fcf51a63c111b8a33d11fccc873014c2b3d6d092158e59a2e3b344b2"}
Nov 02 14:29:06.474089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:06.473941    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2c65b63fcf51a63c111b8a33d11fccc873014c2b3d6d092158e59a2e3b344b2"
Nov 02 14:29:18.034257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.034062    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"]
Nov 02 14:29:18.036267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.036242    5034 cpu_manager.go:414] "containerMap: removing container" podUID="774ec3a9-cda4-4ea3-a028-c23476678fdb" containerName="revoke-keys"
Nov 02 14:29:18.036429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.036418    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="774ec3a9-cda4-4ea3-a028-c23476678fdb" containerName="revoke-keys"
Nov 02 14:29:18.046009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.036823    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="774ec3a9-cda4-4ea3-a028-c23476678fdb" containerName="revoke-keys"
Nov 02 14:29:18.141178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.141114    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.143126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.143095    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"]
Nov 02 14:29:18.158527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.158499    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:29:18.171328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.171305    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd02c09-sczk8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.171451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.171375    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd02c09-sczk8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.171686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.171661    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5j7\" (UniqueName: \"kubernetes.io/projected/9d8b7229-44eb-4f23-b967-24badcb201d8-kube-api-access-dg5j7\") pod \"maas-api-revoke-sub-9fd02c09-sczk8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.172081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.172063    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:29:18.172081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.172075    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:29:18.172182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.172124    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:29:18.272942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.272911    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5j7\" (UniqueName: \"kubernetes.io/projected/9d8b7229-44eb-4f23-b967-24badcb201d8-kube-api-access-dg5j7\") pod \"maas-api-revoke-sub-9fd02c09-sczk8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.273092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.273031    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd02c09-sczk8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.273164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.273114    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd02c09-sczk8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.273928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.273903    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd02c09-sczk8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.275455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.275435    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd02c09-sczk8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.298284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.298230    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5j7\" (UniqueName: \"kubernetes.io/projected/9d8b7229-44eb-4f23-b967-24badcb201d8-kube-api-access-dg5j7\") pod \"maas-api-revoke-sub-9fd02c09-sczk8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.419166 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:29:18.419133    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:29:18.460430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.460400    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:18.779324 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:29:18.779284    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8b7229_44eb_4f23_b967_24badcb201d8.slice/crio-fd0c2b8c6e7e8a5af34163fc9fa336f8fe96bf1f4433b875d565ab5acde612a8 WatchSource:0}: Error finding container fd0c2b8c6e7e8a5af34163fc9fa336f8fe96bf1f4433b875d565ab5acde612a8: Status 404 returned error can't find the container with id fd0c2b8c6e7e8a5af34163fc9fa336f8fe96bf1f4433b875d565ab5acde612a8
Nov 02 14:29:18.784542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:18.784513    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"]
Nov 02 14:29:19.657909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:19.657870    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8" event={"ID":"9d8b7229-44eb-4f23-b967-24badcb201d8","Type":"ContainerStarted","Data":"43da236de8d1880690edd79e36b09fd6a11e183928227c7ab715690dbdb63bf4"}
Nov 02 14:29:19.658300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:19.657952    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8" event={"ID":"9d8b7229-44eb-4f23-b967-24badcb201d8","Type":"ContainerStarted","Data":"fd0c2b8c6e7e8a5af34163fc9fa336f8fe96bf1f4433b875d565ab5acde612a8"}
Nov 02 14:29:19.704428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:19.704358    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8" podStartSLOduration=2.704339856 podStartE2EDuration="2.704339856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:19.700517082 +0000 UTC m=+2750.451821532" watchObservedRunningTime="2026-11-02 14:29:19.704339856 +0000 UTC m=+2750.455644297"
Nov 02 14:29:20.682956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:20.682910    5034 generic.go:356] "Generic (PLEG): container finished" podID="9d8b7229-44eb-4f23-b967-24badcb201d8" containerID="43da236de8d1880690edd79e36b09fd6a11e183928227c7ab715690dbdb63bf4" exitCode=0
Nov 02 14:29:20.683453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:20.683004    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8" event={"ID":"9d8b7229-44eb-4f23-b967-24badcb201d8","Type":"ContainerDied","Data":"43da236de8d1880690edd79e36b09fd6a11e183928227c7ab715690dbdb63bf4"}
Nov 02 14:29:21.875163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:21.875134    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:21.947030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:21.946956    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-openshift-ca\") pod \"9d8b7229-44eb-4f23-b967-24badcb201d8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") "
Nov 02 14:29:21.947174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:21.947125    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8b7229-44eb-4f23-b967-24badcb201d8-kube-api-access-dg5j7\" (UniqueName: \"kubernetes.io/projected/9d8b7229-44eb-4f23-b967-24badcb201d8-kube-api-access-dg5j7\") pod \"9d8b7229-44eb-4f23-b967-24badcb201d8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") "
Nov 02 14:29:21.947236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:21.947180    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-xks-ca\") pod \"9d8b7229-44eb-4f23-b967-24badcb201d8\" (UID: \"9d8b7229-44eb-4f23-b967-24badcb201d8\") "
Nov 02 14:29:21.947371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:21.947349    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-openshift-ca" pod "9d8b7229-44eb-4f23-b967-24badcb201d8" (UID: "9d8b7229-44eb-4f23-b967-24badcb201d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:21.947828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:21.947773    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:21.949700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:21.949659    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8b7229-44eb-4f23-b967-24badcb201d8-kube-api-access-dg5j7" pod "9d8b7229-44eb-4f23-b967-24badcb201d8" (UID: "9d8b7229-44eb-4f23-b967-24badcb201d8"). InnerVolumeSpecName "kube-api-access-dg5j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:21.949988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:21.949948    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-xks-ca" pod "9d8b7229-44eb-4f23-b967-24badcb201d8" (UID: "9d8b7229-44eb-4f23-b967-24badcb201d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:29:22.049545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:22.049517    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg5j7\" (UniqueName: \"kubernetes.io/projected/9d8b7229-44eb-4f23-b967-24badcb201d8-kube-api-access-dg5j7\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:22.049545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:22.049545    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8b7229-44eb-4f23-b967-24badcb201d8-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:22.702714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:22.702680    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8" event={"ID":"9d8b7229-44eb-4f23-b967-24badcb201d8","Type":"ContainerDied","Data":"fd0c2b8c6e7e8a5af34163fc9fa336f8fe96bf1f4433b875d565ab5acde612a8"}
Nov 02 14:29:22.702714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:22.702711    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"
Nov 02 14:29:22.703029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:22.702728    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd0c2b8c6e7e8a5af34163fc9fa336f8fe96bf1f4433b875d565ab5acde612a8"
Nov 02 14:29:32.642195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.642160    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"]
Nov 02 14:29:32.644175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.644121    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9d8b7229-44eb-4f23-b967-24badcb201d8" containerName="revoke-keys"
Nov 02 14:29:32.644370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.644354    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8b7229-44eb-4f23-b967-24badcb201d8" containerName="revoke-keys"
Nov 02 14:29:32.644758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.644737    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8b7229-44eb-4f23-b967-24badcb201d8" containerName="revoke-keys"
Nov 02 14:29:32.650283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.650261    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:32.668107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.668072    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:29:32.682741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.682714    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:29:32.683079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.683059    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:29:32.683234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.683221    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:29:32.721897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.721854    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"]
Nov 02 14:29:32.749827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.740882    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxdm\" (UniqueName: \"kubernetes.io/projected/68c56a91-d36f-4620-a00e-d3fc5948ec15-kube-api-access-fpxdm\") pod \"maas-api-revoke-sub-960ec365-4zzlx\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:32.749827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.741022    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960ec365-4zzlx\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:32.749827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.741302    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960ec365-4zzlx\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:32.868021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.844342    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960ec365-4zzlx\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:32.868021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.844435    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxdm\" (UniqueName: \"kubernetes.io/projected/68c56a91-d36f-4620-a00e-d3fc5948ec15-kube-api-access-fpxdm\") pod \"maas-api-revoke-sub-960ec365-4zzlx\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:32.868021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.844526    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960ec365-4zzlx\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:32.868021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.867654    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-960ec365-4zzlx\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:32.898028 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.897955    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxdm\" (UniqueName: \"kubernetes.io/projected/68c56a91-d36f-4620-a00e-d3fc5948ec15-kube-api-access-fpxdm\") pod \"maas-api-revoke-sub-960ec365-4zzlx\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:32.898975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:32.898948    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-960ec365-4zzlx\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:33.194312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:33.194219    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:33.503659 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:29:33.503624    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c56a91_d36f_4620_a00e_d3fc5948ec15.slice/crio-e4ec5517b287035713d37ba61cf2104394b4cbce8645bc25fa65457a085c120e WatchSource:0}: Error finding container e4ec5517b287035713d37ba61cf2104394b4cbce8645bc25fa65457a085c120e: Status 404 returned error can't find the container with id e4ec5517b287035713d37ba61cf2104394b4cbce8645bc25fa65457a085c120e
Nov 02 14:29:33.506680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:33.506641    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"]
Nov 02 14:29:33.908761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:33.908721    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx" event={"ID":"68c56a91-d36f-4620-a00e-d3fc5948ec15","Type":"ContainerStarted","Data":"188ab232c006cdc461a77e7ba3bbfdadbd186db9eb2a9e98265f00566188570e"}
Nov 02 14:29:33.909264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:33.908805    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx" event={"ID":"68c56a91-d36f-4620-a00e-d3fc5948ec15","Type":"ContainerStarted","Data":"e4ec5517b287035713d37ba61cf2104394b4cbce8645bc25fa65457a085c120e"}
Nov 02 14:29:33.958214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:33.958166    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx" podStartSLOduration=1.958154844 podStartE2EDuration="1.958154844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:33.953193148 +0000 UTC m=+2764.704497589" watchObservedRunningTime="2026-11-02 14:29:33.958154844 +0000 UTC m=+2764.709459282"
Nov 02 14:29:34.919884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:34.919841    5034 generic.go:356] "Generic (PLEG): container finished" podID="68c56a91-d36f-4620-a00e-d3fc5948ec15" containerID="188ab232c006cdc461a77e7ba3bbfdadbd186db9eb2a9e98265f00566188570e" exitCode=0
Nov 02 14:29:34.920377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:34.919925    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx" event={"ID":"68c56a91-d36f-4620-a00e-d3fc5948ec15","Type":"ContainerDied","Data":"188ab232c006cdc461a77e7ba3bbfdadbd186db9eb2a9e98265f00566188570e"}
Nov 02 14:29:36.111158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.111123    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:36.213063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.212982    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c56a91-d36f-4620-a00e-d3fc5948ec15-kube-api-access-fpxdm\" (UniqueName: \"kubernetes.io/projected/68c56a91-d36f-4620-a00e-d3fc5948ec15-kube-api-access-fpxdm\") pod \"68c56a91-d36f-4620-a00e-d3fc5948ec15\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") "
Nov 02 14:29:36.213063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.213042    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-openshift-ca\") pod \"68c56a91-d36f-4620-a00e-d3fc5948ec15\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") "
Nov 02 14:29:36.213423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.213400    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-xks-ca\") pod \"68c56a91-d36f-4620-a00e-d3fc5948ec15\" (UID: \"68c56a91-d36f-4620-a00e-d3fc5948ec15\") "
Nov 02 14:29:36.213882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.213853    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-openshift-ca" pod "68c56a91-d36f-4620-a00e-d3fc5948ec15" (UID: "68c56a91-d36f-4620-a00e-d3fc5948ec15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:36.214204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.214175    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:36.216214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.216184    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-xks-ca" pod "68c56a91-d36f-4620-a00e-d3fc5948ec15" (UID: "68c56a91-d36f-4620-a00e-d3fc5948ec15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:29:36.219320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.219293    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c56a91-d36f-4620-a00e-d3fc5948ec15-kube-api-access-fpxdm" pod "68c56a91-d36f-4620-a00e-d3fc5948ec15" (UID: "68c56a91-d36f-4620-a00e-d3fc5948ec15"). InnerVolumeSpecName "kube-api-access-fpxdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:36.316383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.316346    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c56a91-d36f-4620-a00e-d3fc5948ec15-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:36.316383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.316376    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpxdm\" (UniqueName: \"kubernetes.io/projected/68c56a91-d36f-4620-a00e-d3fc5948ec15-kube-api-access-fpxdm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:36.938361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.938322    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx" event={"ID":"68c56a91-d36f-4620-a00e-d3fc5948ec15","Type":"ContainerDied","Data":"e4ec5517b287035713d37ba61cf2104394b4cbce8645bc25fa65457a085c120e"}
Nov 02 14:29:36.938550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.938374    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4ec5517b287035713d37ba61cf2104394b4cbce8645bc25fa65457a085c120e"
Nov 02 14:29:36.938550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:36.938336    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"
Nov 02 14:29:44.747800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.741295    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"]
Nov 02 14:29:44.757822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.754426    5034 cpu_manager.go:414] "containerMap: removing container" podUID="68c56a91-d36f-4620-a00e-d3fc5948ec15" containerName="revoke-keys"
Nov 02 14:29:44.757822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.754456    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c56a91-d36f-4620-a00e-d3fc5948ec15" containerName="revoke-keys"
Nov 02 14:29:44.757822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.754708    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c56a91-d36f-4620-a00e-d3fc5948ec15" containerName="revoke-keys"
Nov 02 14:29:44.779801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.770761    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:44.816903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.816842    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"]
Nov 02 14:29:44.840802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.832223    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:29:44.840802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.832365    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:29:44.840802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.832369    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:29:44.840802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.832475    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:29:44.876772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.870689    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f05c5d36-462j8\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:44.876772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.870767    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8vc6\" (UniqueName: \"kubernetes.io/projected/8024c587-a435-4718-89d3-c461d97aeb18-kube-api-access-b8vc6\") pod \"maas-api-revoke-sub-f05c5d36-462j8\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:44.876772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.870812    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f05c5d36-462j8\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:44.972529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.972494    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8vc6\" (UniqueName: \"kubernetes.io/projected/8024c587-a435-4718-89d3-c461d97aeb18-kube-api-access-b8vc6\") pod \"maas-api-revoke-sub-f05c5d36-462j8\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:44.972721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.972540    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f05c5d36-462j8\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:44.972867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.972844    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f05c5d36-462j8\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:44.973296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.973269    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f05c5d36-462j8\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:44.975141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:44.975120    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f05c5d36-462j8\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:45.001514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:45.001450    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8vc6\" (UniqueName: \"kubernetes.io/projected/8024c587-a435-4718-89d3-c461d97aeb18-kube-api-access-b8vc6\") pod \"maas-api-revoke-sub-f05c5d36-462j8\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:45.178741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:45.178713    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:45.492338 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:29:45.492281    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8024c587_a435_4718_89d3_c461d97aeb18.slice/crio-b6b803e8437cb00fc615b35c12374c81b477be7c53db1c9c5ec2186c1e12dcfc WatchSource:0}: Error finding container b6b803e8437cb00fc615b35c12374c81b477be7c53db1c9c5ec2186c1e12dcfc: Status 404 returned error can't find the container with id b6b803e8437cb00fc615b35c12374c81b477be7c53db1c9c5ec2186c1e12dcfc
Nov 02 14:29:45.493763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:45.493734    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"]
Nov 02 14:29:46.024772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:46.024740    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8" event={"ID":"8024c587-a435-4718-89d3-c461d97aeb18","Type":"ContainerStarted","Data":"d89e24d0fa1e31af34282e792825c5ea5bb76fd958bae7bdbbcfb8f5fc82eb47"}
Nov 02 14:29:46.025274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:46.024810    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8" event={"ID":"8024c587-a435-4718-89d3-c461d97aeb18","Type":"ContainerStarted","Data":"b6b803e8437cb00fc615b35c12374c81b477be7c53db1c9c5ec2186c1e12dcfc"}
Nov 02 14:29:46.099705 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:46.099654    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8" podStartSLOduration=2.099641599 podStartE2EDuration="2.099641599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:29:46.097574458 +0000 UTC m=+2776.848878896" watchObservedRunningTime="2026-11-02 14:29:46.099641599 +0000 UTC m=+2776.850946036"
Nov 02 14:29:47.034947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:47.034915    5034 generic.go:356] "Generic (PLEG): container finished" podID="8024c587-a435-4718-89d3-c461d97aeb18" containerID="d89e24d0fa1e31af34282e792825c5ea5bb76fd958bae7bdbbcfb8f5fc82eb47" exitCode=0
Nov 02 14:29:47.035376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:47.034966    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8" event={"ID":"8024c587-a435-4718-89d3-c461d97aeb18","Type":"ContainerDied","Data":"d89e24d0fa1e31af34282e792825c5ea5bb76fd958bae7bdbbcfb8f5fc82eb47"}
Nov 02 14:29:48.247300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.247274    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:48.350279 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.350192    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8024c587-a435-4718-89d3-c461d97aeb18-kube-api-access-b8vc6\" (UniqueName: \"kubernetes.io/projected/8024c587-a435-4718-89d3-c461d97aeb18-kube-api-access-b8vc6\") pod \"8024c587-a435-4718-89d3-c461d97aeb18\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") "
Nov 02 14:29:48.350448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.350335    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-xks-ca\") pod \"8024c587-a435-4718-89d3-c461d97aeb18\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") "
Nov 02 14:29:48.351481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.350765    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-openshift-ca\") pod \"8024c587-a435-4718-89d3-c461d97aeb18\" (UID: \"8024c587-a435-4718-89d3-c461d97aeb18\") "
Nov 02 14:29:48.352268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.351417    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-openshift-ca" pod "8024c587-a435-4718-89d3-c461d97aeb18" (UID: "8024c587-a435-4718-89d3-c461d97aeb18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:29:48.352933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.352872    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-xks-ca" pod "8024c587-a435-4718-89d3-c461d97aeb18" (UID: "8024c587-a435-4718-89d3-c461d97aeb18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:29:48.353336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.353310    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8024c587-a435-4718-89d3-c461d97aeb18-kube-api-access-b8vc6" pod "8024c587-a435-4718-89d3-c461d97aeb18" (UID: "8024c587-a435-4718-89d3-c461d97aeb18"). InnerVolumeSpecName "kube-api-access-b8vc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:29:48.414736 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:29:48.414692    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:29:48.453594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.453566    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8vc6\" (UniqueName: \"kubernetes.io/projected/8024c587-a435-4718-89d3-c461d97aeb18-kube-api-access-b8vc6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:48.453594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.453593    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:48.453773 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:48.453602    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8024c587-a435-4718-89d3-c461d97aeb18-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:29:49.066799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:49.054950    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8" event={"ID":"8024c587-a435-4718-89d3-c461d97aeb18","Type":"ContainerDied","Data":"b6b803e8437cb00fc615b35c12374c81b477be7c53db1c9c5ec2186c1e12dcfc"}
Nov 02 14:29:49.066799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:49.055083    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6b803e8437cb00fc615b35c12374c81b477be7c53db1c9c5ec2186c1e12dcfc"
Nov 02 14:29:49.066799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:49.055174    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"
Nov 02 14:29:58.561364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.561322    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"]
Nov 02 14:29:58.562263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.562245    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8024c587-a435-4718-89d3-c461d97aeb18" containerName="revoke-keys"
Nov 02 14:29:58.562326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.562266    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8024c587-a435-4718-89d3-c461d97aeb18" containerName="revoke-keys"
Nov 02 14:29:58.562430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.562420    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8024c587-a435-4718-89d3-c461d97aeb18" containerName="revoke-keys"
Nov 02 14:29:58.566363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.566347    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.609747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.609534    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:29:58.612321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.610336    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:29:58.612321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.611133    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:29:58.612321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.611911    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:29:58.646629 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.646588    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"]
Nov 02 14:29:58.693004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.692625    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa18ab9a-d24t9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.693004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.692686    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa18ab9a-d24t9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.693004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.692843    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbn2x\" (UniqueName: \"kubernetes.io/projected/04c74a77-d231-4dc7-a08a-04110d047bd9-kube-api-access-sbn2x\") pod \"maas-api-revoke-sub-aa18ab9a-d24t9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.794055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.794010    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa18ab9a-d24t9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.794244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.794155    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbn2x\" (UniqueName: \"kubernetes.io/projected/04c74a77-d231-4dc7-a08a-04110d047bd9-kube-api-access-sbn2x\") pod \"maas-api-revoke-sub-aa18ab9a-d24t9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.794448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.794425    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa18ab9a-d24t9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.795711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.795497    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa18ab9a-d24t9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.797606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.797580    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa18ab9a-d24t9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.822486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.822414    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbn2x\" (UniqueName: \"kubernetes.io/projected/04c74a77-d231-4dc7-a08a-04110d047bd9-kube-api-access-sbn2x\") pod \"maas-api-revoke-sub-aa18ab9a-d24t9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:58.884051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:58.884020    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:29:59.240333 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:29:59.240304    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c74a77_d231_4dc7_a08a_04110d047bd9.slice/crio-0abda4fb4b94c52518bf5f8a55f3f064bbe9ccea843d9243041145b5e3d5aceb WatchSource:0}: Error finding container 0abda4fb4b94c52518bf5f8a55f3f064bbe9ccea843d9243041145b5e3d5aceb: Status 404 returned error can't find the container with id 0abda4fb4b94c52518bf5f8a55f3f064bbe9ccea843d9243041145b5e3d5aceb
Nov 02 14:29:59.241853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:29:59.241823    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"]
Nov 02 14:30:00.167999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.167962    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"]
Nov 02 14:30:00.174104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.174078    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:00.187761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.187690    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"]
Nov 02 14:30:00.195646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.195622    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"]
Nov 02 14:30:00.195828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.195765    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:00.202381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.202357    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"]
Nov 02 14:30:00.202646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.202613    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:00.210009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.209989    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"]
Nov 02 14:30:00.210237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.210217    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:00.226825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.226460    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"]
Nov 02 14:30:00.226825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.226564    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.240813    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtq7s\" (UniqueName: \"kubernetes.io/projected/00f54dd3-b21f-4705-af6f-d29c35b9368c-kube-api-access-vtq7s\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48\" (UID: \"00f54dd3-b21f-4705-af6f-d29c35b9368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.240918    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x265m\" (UniqueName: \"kubernetes.io/projected/6d7b00a2-30e2-4c1f-bc50-d98158707200-kube-api-access-x265m\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7\" (UID: \"6d7b00a2-30e2-4c1f-bc50-d98158707200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.240966    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b00a2-30e2-4c1f-bc50-d98158707200-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7\" (UID: \"6d7b00a2-30e2-4c1f-bc50-d98158707200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.241038    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh\" (UID: \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.241121    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t52w5\" (UniqueName: \"kubernetes.io/projected/12c10b34-f265-434c-a511-077efe1688d0-kube-api-access-t52w5\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb\" (UID: \"12c10b34-f265-434c-a511-077efe1688d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.241175    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798e455b-975d-468b-bc10-3e020e992626-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl\" (UID: \"798e455b-975d-468b-bc10-3e020e992626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.241213    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgcbm\" (UniqueName: \"kubernetes.io/projected/798e455b-975d-468b-bc10-3e020e992626-kube-api-access-bgcbm\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl\" (UID: \"798e455b-975d-468b-bc10-3e020e992626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.241280    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8r5\" (UniqueName: \"kubernetes.io/projected/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-kube-api-access-wt8r5\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh\" (UID: \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.241311    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f54dd3-b21f-4705-af6f-d29c35b9368c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48\" (UID: \"00f54dd3-b21f-4705-af6f-d29c35b9368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.241370    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c10b34-f265-434c-a511-077efe1688d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb\" (UID: \"12c10b34-f265-434c-a511-077efe1688d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.245081    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9" event={"ID":"04c74a77-d231-4dc7-a08a-04110d047bd9","Type":"ContainerStarted","Data":"ee940b3a2a8303c875cfcef0822a96db4116e472b698e2c5e89f358420b3787a"}
Nov 02 14:30:00.253020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.245184    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9" event={"ID":"04c74a77-d231-4dc7-a08a-04110d047bd9","Type":"ContainerStarted","Data":"0abda4fb4b94c52518bf5f8a55f3f064bbe9ccea843d9243041145b5e3d5aceb"}
Nov 02 14:30:00.264846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.254392    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"]
Nov 02 14:30:00.268152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.268121    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"]
Nov 02 14:30:00.279820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.279595    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"]
Nov 02 14:30:00.279820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.279744    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:00.286997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.285234    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"]
Nov 02 14:30:00.299802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.299441    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"]
Nov 02 14:30:00.299802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.299487    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"]
Nov 02 14:30:00.299802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.299579    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:00.306805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.306019    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"]
Nov 02 14:30:00.326798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.326358    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352262    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x265m\" (UniqueName: \"kubernetes.io/projected/6d7b00a2-30e2-4c1f-bc50-d98158707200-kube-api-access-x265m\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7\" (UID: \"6d7b00a2-30e2-4c1f-bc50-d98158707200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352295    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"]
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352312    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b00a2-30e2-4c1f-bc50-d98158707200-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7\" (UID: \"6d7b00a2-30e2-4c1f-bc50-d98158707200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352400    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh\" (UID: \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352454    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvwm\" (UniqueName: \"kubernetes.io/projected/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-kube-api-access-9wvwm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd\" (UID: \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352534    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd\" (UID: \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352569    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t52w5\" (UniqueName: \"kubernetes.io/projected/12c10b34-f265-434c-a511-077efe1688d0-kube-api-access-t52w5\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb\" (UID: \"12c10b34-f265-434c-a511-077efe1688d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352663    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798e455b-975d-468b-bc10-3e020e992626-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl\" (UID: \"798e455b-975d-468b-bc10-3e020e992626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352709    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcbm\" (UniqueName: \"kubernetes.io/projected/798e455b-975d-468b-bc10-3e020e992626-kube-api-access-bgcbm\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl\" (UID: \"798e455b-975d-468b-bc10-3e020e992626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352772    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8r5\" (UniqueName: \"kubernetes.io/projected/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-kube-api-access-wt8r5\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh\" (UID: \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352833    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-kube-api-access-lxbcf\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c\" (UID: \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352864    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f54dd3-b21f-4705-af6f-d29c35b9368c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48\" (UID: \"00f54dd3-b21f-4705-af6f-d29c35b9368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352915    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c\" (UID: \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.352986    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110008e0-3d52-4923-9188-98bafbfccf54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj\" (UID: \"110008e0-3d52-4923-9188-98bafbfccf54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.353043    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c10b34-f265-434c-a511-077efe1688d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb\" (UID: \"12c10b34-f265-434c-a511-077efe1688d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.353094    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dkq\" (UniqueName: \"kubernetes.io/projected/110008e0-3d52-4923-9188-98bafbfccf54-kube-api-access-z4dkq\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj\" (UID: \"110008e0-3d52-4923-9188-98bafbfccf54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.353289    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtq7s\" (UniqueName: \"kubernetes.io/projected/00f54dd3-b21f-4705-af6f-d29c35b9368c-kube-api-access-vtq7s\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48\" (UID: \"00f54dd3-b21f-4705-af6f-d29c35b9368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.353358    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh\" (UID: \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.355002    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f54dd3-b21f-4705-af6f-d29c35b9368c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48\" (UID: \"00f54dd3-b21f-4705-af6f-d29c35b9368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.355380    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798e455b-975d-468b-bc10-3e020e992626-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl\" (UID: \"798e455b-975d-468b-bc10-3e020e992626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.357222    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c10b34-f265-434c-a511-077efe1688d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb\" (UID: \"12c10b34-f265-434c-a511-077efe1688d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:00.382601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.358020    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b00a2-30e2-4c1f-bc50-d98158707200-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7\" (UID: \"6d7b00a2-30e2-4c1f-bc50-d98158707200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:00.437488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.437012    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x265m\" (UniqueName: \"kubernetes.io/projected/6d7b00a2-30e2-4c1f-bc50-d98158707200-kube-api-access-x265m\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7\" (UID: \"6d7b00a2-30e2-4c1f-bc50-d98158707200\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:00.442827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.442563    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"]
Nov 02 14:30:00.456280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.456182    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t52w5\" (UniqueName: \"kubernetes.io/projected/12c10b34-f265-434c-a511-077efe1688d0-kube-api-access-t52w5\") pod \"maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb\" (UID: \"12c10b34-f265-434c-a511-077efe1688d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:00.466490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.466178    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvwm\" (UniqueName: \"kubernetes.io/projected/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-kube-api-access-9wvwm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd\" (UID: \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:00.466490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.466289    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd\" (UID: \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:00.474491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.467247    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd\" (UID: \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:00.474491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.467919    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-kube-api-access-lxbcf\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c\" (UID: \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:00.474491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.468336    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c\" (UID: \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:00.474491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.468396    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110008e0-3d52-4923-9188-98bafbfccf54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj\" (UID: \"110008e0-3d52-4923-9188-98bafbfccf54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:00.474491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.468807    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"]
Nov 02 14:30:00.474491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.470303    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c\" (UID: \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:00.474491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.471421    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110008e0-3d52-4923-9188-98bafbfccf54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj\" (UID: \"110008e0-3d52-4923-9188-98bafbfccf54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:00.474491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.471998    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dkq\" (UniqueName: \"kubernetes.io/projected/110008e0-3d52-4923-9188-98bafbfccf54-kube-api-access-z4dkq\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj\" (UID: \"110008e0-3d52-4923-9188-98bafbfccf54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:00.494545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.493501    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8r5\" (UniqueName: \"kubernetes.io/projected/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-kube-api-access-wt8r5\") pod \"maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh\" (UID: \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:00.512337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.512300    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvwm\" (UniqueName: \"kubernetes.io/projected/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-kube-api-access-9wvwm\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd\" (UID: \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:00.513319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.512953    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtq7s\" (UniqueName: \"kubernetes.io/projected/00f54dd3-b21f-4705-af6f-d29c35b9368c-kube-api-access-vtq7s\") pod \"maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48\" (UID: \"00f54dd3-b21f-4705-af6f-d29c35b9368c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:00.515297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.515066    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:00.520809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.520756    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgcbm\" (UniqueName: \"kubernetes.io/projected/798e455b-975d-468b-bc10-3e020e992626-kube-api-access-bgcbm\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl\" (UID: \"798e455b-975d-468b-bc10-3e020e992626\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:00.524470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.524400    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"]
Nov 02 14:30:00.524703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.524676    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:00.529594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.529572    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:00.534749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.534702    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-kube-api-access-lxbcf\") pod \"maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c\" (UID: \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:00.539811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.538541    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dkq\" (UniqueName: \"kubernetes.io/projected/110008e0-3d52-4923-9188-98bafbfccf54-kube-api-access-z4dkq\") pod \"maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj\" (UID: \"110008e0-3d52-4923-9188-98bafbfccf54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:00.550993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.544413    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:00.550993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.546506    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"]
Nov 02 14:30:00.567131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.565970    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"]
Nov 02 14:30:00.569909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.567635    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:00.577040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.576109    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce138676-ad2e-4761-ad72-571e0b3bf23c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw\" (UID: \"ce138676-ad2e-4761-ad72-571e0b3bf23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:00.577040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.576315    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnbvw\" (UniqueName: \"kubernetes.io/projected/ce138676-ad2e-4761-ad72-571e0b3bf23c-kube-api-access-hnbvw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw\" (UID: \"ce138676-ad2e-4761-ad72-571e0b3bf23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:00.577424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.577392    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:00.582168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.581721    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"]
Nov 02 14:30:00.586543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.586458    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:00.593544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.593524    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:00.594588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.594509    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"]
Nov 02 14:30:00.607344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.607316    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"]
Nov 02 14:30:00.612462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.612443    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:00.627819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.627794    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"]
Nov 02 14:30:00.630868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.629724    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:00.635866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.635767    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:00.640007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.638290    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"]
Nov 02 14:30:00.640007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.638335    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"]
Nov 02 14:30:00.647038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.646967    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:00.684223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.678037    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljrt2\" (UniqueName: \"kubernetes.io/projected/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-kube-api-access-ljrt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb\" (UID: \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:00.684223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.678174    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c703d-dd22-4a0b-a2c0-93e2ac208979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9\" (UID: \"732c703d-dd22-4a0b-a2c0-93e2ac208979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:00.684223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.678303    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmpdq\" (UniqueName: \"kubernetes.io/projected/732c703d-dd22-4a0b-a2c0-93e2ac208979-kube-api-access-qmpdq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9\" (UID: \"732c703d-dd22-4a0b-a2c0-93e2ac208979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:00.684223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.678339    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tmd\" (UniqueName: \"kubernetes.io/projected/786d86b7-0109-4fe1-abce-3960750cc80d-kube-api-access-v6tmd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6\" (UID: \"786d86b7-0109-4fe1-abce-3960750cc80d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:00.684223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.678394    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce138676-ad2e-4761-ad72-571e0b3bf23c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw\" (UID: \"ce138676-ad2e-4761-ad72-571e0b3bf23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:00.684223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.678438    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c667d91-1fd4-438c-b3a3-8281cf003b88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg\" (UID: \"1c667d91-1fd4-438c-b3a3-8281cf003b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:00.701645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.697606    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8wx6\" (UniqueName: \"kubernetes.io/projected/1c667d91-1fd4-438c-b3a3-8281cf003b88-kube-api-access-g8wx6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg\" (UID: \"1c667d91-1fd4-438c-b3a3-8281cf003b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:00.701645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.697707    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnbvw\" (UniqueName: \"kubernetes.io/projected/ce138676-ad2e-4761-ad72-571e0b3bf23c-kube-api-access-hnbvw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw\" (UID: \"ce138676-ad2e-4761-ad72-571e0b3bf23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:00.701645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.697847    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786d86b7-0109-4fe1-abce-3960750cc80d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6\" (UID: \"786d86b7-0109-4fe1-abce-3960750cc80d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:00.701645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.697912    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb\" (UID: \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:00.701645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.699613    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce138676-ad2e-4761-ad72-571e0b3bf23c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw\" (UID: \"ce138676-ad2e-4761-ad72-571e0b3bf23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:00.751423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.751376    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"]
Nov 02 14:30:00.751841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.751741    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:00.759144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.759115    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnbvw\" (UniqueName: \"kubernetes.io/projected/ce138676-ad2e-4761-ad72-571e0b3bf23c-kube-api-access-hnbvw\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw\" (UID: \"ce138676-ad2e-4761-ad72-571e0b3bf23c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:00.760180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.760156    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:00.766249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.765519    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"]
Nov 02 14:30:00.786139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.786102    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:00.804591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.790547    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"]
Nov 02 14:30:00.804591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.801870    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"]
Nov 02 14:30:00.815283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.812287    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"]
Nov 02 14:30:00.820816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.818863    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2j2h\" (UniqueName: \"kubernetes.io/projected/9a96c9cc-4506-402a-8923-620b9381ecc9-kube-api-access-s2j2h\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg\" (UID: \"9a96c9cc-4506-402a-8923-620b9381ecc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:00.820816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.818930    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c703d-dd22-4a0b-a2c0-93e2ac208979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9\" (UID: \"732c703d-dd22-4a0b-a2c0-93e2ac208979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:00.820816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.819052    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmpdq\" (UniqueName: \"kubernetes.io/projected/732c703d-dd22-4a0b-a2c0-93e2ac208979-kube-api-access-qmpdq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9\" (UID: \"732c703d-dd22-4a0b-a2c0-93e2ac208979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:00.820816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.819089    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tmd\" (UniqueName: \"kubernetes.io/projected/786d86b7-0109-4fe1-abce-3960750cc80d-kube-api-access-v6tmd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6\" (UID: \"786d86b7-0109-4fe1-abce-3960750cc80d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:00.820816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.819165    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c667d91-1fd4-438c-b3a3-8281cf003b88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg\" (UID: \"1c667d91-1fd4-438c-b3a3-8281cf003b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:00.820816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.819323    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a96c9cc-4506-402a-8923-620b9381ecc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg\" (UID: \"9a96c9cc-4506-402a-8923-620b9381ecc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:00.820816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.819377    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wx6\" (UniqueName: \"kubernetes.io/projected/1c667d91-1fd4-438c-b3a3-8281cf003b88-kube-api-access-g8wx6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg\" (UID: \"1c667d91-1fd4-438c-b3a3-8281cf003b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:00.820816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.819502    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786d86b7-0109-4fe1-abce-3960750cc80d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6\" (UID: \"786d86b7-0109-4fe1-abce-3960750cc80d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:00.820816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.819564    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb\" (UID: \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:00.823659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.823630    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljrt2\" (UniqueName: \"kubernetes.io/projected/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-kube-api-access-ljrt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb\" (UID: \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:00.825215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.825188    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c667d91-1fd4-438c-b3a3-8281cf003b88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg\" (UID: \"1c667d91-1fd4-438c-b3a3-8281cf003b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:00.826182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.826155    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786d86b7-0109-4fe1-abce-3960750cc80d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6\" (UID: \"786d86b7-0109-4fe1-abce-3960750cc80d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:00.827315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.827043    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb\" (UID: \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:00.827315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.827263    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9" podStartSLOduration=2.827244845 podStartE2EDuration="2.827244845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:00.519161146 +0000 UTC m=+2791.270465593" watchObservedRunningTime="2026-11-02 14:30:00.827244845 +0000 UTC m=+2791.578549285"
Nov 02 14:30:00.827807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.827764    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c703d-dd22-4a0b-a2c0-93e2ac208979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9\" (UID: \"732c703d-dd22-4a0b-a2c0-93e2ac208979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:00.827889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.827836    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"]
Nov 02 14:30:00.872379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.830297    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:00.872379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.852833    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"]
Nov 02 14:30:00.872379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.860349    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljrt2\" (UniqueName: \"kubernetes.io/projected/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-kube-api-access-ljrt2\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb\" (UID: \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:00.872379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.864372    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmpdq\" (UniqueName: \"kubernetes.io/projected/732c703d-dd22-4a0b-a2c0-93e2ac208979-kube-api-access-qmpdq\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9\" (UID: \"732c703d-dd22-4a0b-a2c0-93e2ac208979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:00.872379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.865099    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tmd\" (UniqueName: \"kubernetes.io/projected/786d86b7-0109-4fe1-abce-3960750cc80d-kube-api-access-v6tmd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6\" (UID: \"786d86b7-0109-4fe1-abce-3960750cc80d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:00.872379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.867634    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wx6\" (UniqueName: \"kubernetes.io/projected/1c667d91-1fd4-438c-b3a3-8281cf003b88-kube-api-access-g8wx6\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg\" (UID: \"1c667d91-1fd4-438c-b3a3-8281cf003b88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:00.926426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.926372    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2cv\" (UniqueName: \"kubernetes.io/projected/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-kube-api-access-ld2cv\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4\" (UID: \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:00.926818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.926792    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5391b42a-6255-4531-957f-31dd8616f975-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2\" (UID: \"5391b42a-6255-4531-957f-31dd8616f975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:00.927106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.927074    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2j2h\" (UniqueName: \"kubernetes.io/projected/9a96c9cc-4506-402a-8923-620b9381ecc9-kube-api-access-s2j2h\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg\" (UID: \"9a96c9cc-4506-402a-8923-620b9381ecc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:00.927480 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.927456    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4\" (UID: \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:00.927617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.927600    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a96c9cc-4506-402a-8923-620b9381ecc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg\" (UID: \"9a96c9cc-4506-402a-8923-620b9381ecc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:00.927762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.927746    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdnc\" (UniqueName: \"kubernetes.io/projected/5391b42a-6255-4531-957f-31dd8616f975-kube-api-access-qwdnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2\" (UID: \"5391b42a-6255-4531-957f-31dd8616f975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:00.929651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.929628    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a96c9cc-4506-402a-8923-620b9381ecc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg\" (UID: \"9a96c9cc-4506-402a-8923-620b9381ecc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:00.937612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.937246    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:00.958188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:00.958132    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2j2h\" (UniqueName: \"kubernetes.io/projected/9a96c9cc-4506-402a-8923-620b9381ecc9-kube-api-access-s2j2h\") pod \"maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg\" (UID: \"9a96c9cc-4506-402a-8923-620b9381ecc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:01.060802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.036049    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4\" (UID: \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:01.143945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.143905    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdnc\" (UniqueName: \"kubernetes.io/projected/5391b42a-6255-4531-957f-31dd8616f975-kube-api-access-qwdnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2\" (UID: \"5391b42a-6255-4531-957f-31dd8616f975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:01.144249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.144233    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2cv\" (UniqueName: \"kubernetes.io/projected/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-kube-api-access-ld2cv\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4\" (UID: \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:01.144477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.144464    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5391b42a-6255-4531-957f-31dd8616f975-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2\" (UID: \"5391b42a-6255-4531-957f-31dd8616f975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:01.144841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.048142    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:01.151955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.049196    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4\" (UID: \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:01.151955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.111241    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:01.151955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.139220    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:01.151955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.145744    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5391b42a-6255-4531-957f-31dd8616f975-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2\" (UID: \"5391b42a-6255-4531-957f-31dd8616f975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:01.260813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.257215    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdnc\" (UniqueName: \"kubernetes.io/projected/5391b42a-6255-4531-957f-31dd8616f975-kube-api-access-qwdnc\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2\" (UID: \"5391b42a-6255-4531-957f-31dd8616f975\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:01.270768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.263056    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:01.425283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.425004    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:01.455569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.455495    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2cv\" (UniqueName: \"kubernetes.io/projected/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-kube-api-access-ld2cv\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4\" (UID: \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:01.598583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:01.598536    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:02.550112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:02.550017    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:02.569844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:02.569808    5034 generic.go:356] "Generic (PLEG): container finished" podID="04c74a77-d231-4dc7-a08a-04110d047bd9" containerID="ee940b3a2a8303c875cfcef0822a96db4116e472b698e2c5e89f358420b3787a" exitCode=0
Nov 02 14:30:02.707473 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:02.706353    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c10b34_f265_434c_a511_077efe1688d0.slice/crio-2b17cd242825b082418d5eaced9a348545f189e1328fc3ce50c1414a88d72484 WatchSource:0}: Error finding container 2b17cd242825b082418d5eaced9a348545f189e1328fc3ce50c1414a88d72484: Status 404 returned error can't find the container with id 2b17cd242825b082418d5eaced9a348545f189e1328fc3ce50c1414a88d72484
Nov 02 14:30:02.716879 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:02.716248    5034 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.267s"
Nov 02 14:30:02.716879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:02.716287    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9" event={"ID":"04c74a77-d231-4dc7-a08a-04110d047bd9","Type":"ContainerDied","Data":"ee940b3a2a8303c875cfcef0822a96db4116e472b698e2c5e89f358420b3787a"}
Nov 02 14:30:02.716879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:02.716467    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"]
Nov 02 14:30:02.810652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:02.810276    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"]
Nov 02 14:30:02.847822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:02.841632    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"]
Nov 02 14:30:02.870016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:02.869976    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"]
Nov 02 14:30:03.091606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.090060    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"]
Nov 02 14:30:03.104997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.104958    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"]
Nov 02 14:30:03.141885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.141832    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"]
Nov 02 14:30:03.148249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.148192    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"]
Nov 02 14:30:03.166448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.166412    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"]
Nov 02 14:30:03.185886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.185855    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"]
Nov 02 14:30:03.337532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.337500    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"]
Nov 02 14:30:03.348115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.348081    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"]
Nov 02 14:30:03.358828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.358796    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"]
Nov 02 14:30:03.363970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.363942    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"]
Nov 02 14:30:03.406709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.406682    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"]
Nov 02 14:30:03.415198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.415165    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"]
Nov 02 14:30:03.474439 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:03.474394    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5267a8a7_bd52_4e1b_87eb_d6c4085e0769.slice/crio-92f610a6eade6b1f9f9d738dcda9cb384a02645fe760fc93e9607887d3502428 WatchSource:0}: Error finding container 92f610a6eade6b1f9f9d738dcda9cb384a02645fe760fc93e9607887d3502428: Status 404 returned error can't find the container with id 92f610a6eade6b1f9f9d738dcda9cb384a02645fe760fc93e9607887d3502428
Nov 02 14:30:03.592202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.592166    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" event={"ID":"798e455b-975d-468b-bc10-3e020e992626","Type":"ContainerStarted","Data":"3d0c265ae495fd801d8411c114f85497073bc5e501b33e96ea46251c732e8deb"}
Nov 02 14:30:03.592662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.592229    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" event={"ID":"798e455b-975d-468b-bc10-3e020e992626","Type":"ContainerStarted","Data":"e0d225d74e5a8b7eb01e7b7275768d8eec355b553ded8c6286c3cbc8d99f9ba7"}
Nov 02 14:30:03.594000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.593975    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" event={"ID":"00f54dd3-b21f-4705-af6f-d29c35b9368c","Type":"ContainerStarted","Data":"159572d135b1d6f3cabfaf033ff8dc95ad5d3d6b5645c15e9eacef281184e624"}
Nov 02 14:30:03.595453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.595429    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" event={"ID":"5267a8a7-bd52-4e1b-87eb-d6c4085e0769","Type":"ContainerStarted","Data":"92f610a6eade6b1f9f9d738dcda9cb384a02645fe760fc93e9607887d3502428"}
Nov 02 14:30:03.600086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.600055    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" event={"ID":"110008e0-3d52-4923-9188-98bafbfccf54","Type":"ContainerStarted","Data":"a4ff98dade7571c0572fbc541f627d9ae40611407e2efe00261540a94d26b9a1"}
Nov 02 14:30:03.610203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.610164    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" event={"ID":"786d86b7-0109-4fe1-abce-3960750cc80d","Type":"ContainerStarted","Data":"067a67f9edb01ffff5f5cfe3937c67ea9e1bf092e08e2185fc7e8e8395a1b25e"}
Nov 02 14:30:03.614305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.613905    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" event={"ID":"1c6427b3-906d-47bc-938f-c38f0c1b5b5a","Type":"ContainerStarted","Data":"90fbb95cbba1de89194c7f1f021da577be8a27b92dff26207750632f5646466c"}
Nov 02 14:30:03.617711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.617676    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" event={"ID":"12c10b34-f265-434c-a511-077efe1688d0","Type":"ContainerStarted","Data":"2b17cd242825b082418d5eaced9a348545f189e1328fc3ce50c1414a88d72484"}
Nov 02 14:30:03.619375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.619346    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" event={"ID":"a668bbd8-8a19-42f0-b635-7a88c4cd6b35","Type":"ContainerStarted","Data":"deb9911ddfd072a63803a852093b49af0cc70f870e1df0bd5a340e26259d64a0"}
Nov 02 14:30:03.621172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.621145    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" event={"ID":"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68","Type":"ContainerStarted","Data":"b861818721e0f0233512546415b90bf26a7d2d5a9fe027366357cb8cbb7fd16a"}
Nov 02 14:30:03.624157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.622891    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" event={"ID":"ce138676-ad2e-4761-ad72-571e0b3bf23c","Type":"ContainerStarted","Data":"fc25122adf47f8d5a3a785c6c84433af27d21040f67c33cfd5ac72b26901d847"}
Nov 02 14:30:03.624295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.624270    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" event={"ID":"6d7b00a2-30e2-4c1f-bc50-d98158707200","Type":"ContainerStarted","Data":"5ecade2a1f02c5f3b9ea0387c772af9656a402198dc2afbb37eca09d11b0e871"}
Nov 02 14:30:03.641358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:03.641300    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" podStartSLOduration=3.641285915 podStartE2EDuration="3.641285915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:03.636089717 +0000 UTC m=+2794.387394155" watchObservedRunningTime="2026-11-02 14:30:03.641285915 +0000 UTC m=+2794.392590386"
Nov 02 14:30:03.672528 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:03.672487    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5391b42a_6255_4531_957f_31dd8616f975.slice/crio-4ba183242cd63eced2ec93e2dd136bcd0642df123b0525680395ff667d2b6580 WatchSource:0}: Error finding container 4ba183242cd63eced2ec93e2dd136bcd0642df123b0525680395ff667d2b6580: Status 404 returned error can't find the container with id 4ba183242cd63eced2ec93e2dd136bcd0642df123b0525680395ff667d2b6580
Nov 02 14:30:03.678326 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:03.678290    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod732c703d_dd22_4a0b_a2c0_93e2ac208979.slice/crio-4360ed860c9654956784e37dede48af7574639c7e57e5156ec277e047c31626e WatchSource:0}: Error finding container 4360ed860c9654956784e37dede48af7574639c7e57e5156ec277e047c31626e: Status 404 returned error can't find the container with id 4360ed860c9654956784e37dede48af7574639c7e57e5156ec277e047c31626e
Nov 02 14:30:04.337102 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.337057    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:30:04.456635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.456596    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04c74a77-d231-4dc7-a08a-04110d047bd9-kube-api-access-sbn2x\" (UniqueName: \"kubernetes.io/projected/04c74a77-d231-4dc7-a08a-04110d047bd9-kube-api-access-sbn2x\") pod \"04c74a77-d231-4dc7-a08a-04110d047bd9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") "
Nov 02 14:30:04.456886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.456740    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-openshift-ca\") pod \"04c74a77-d231-4dc7-a08a-04110d047bd9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") "
Nov 02 14:30:04.456976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.456962    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-xks-ca\") pod \"04c74a77-d231-4dc7-a08a-04110d047bd9\" (UID: \"04c74a77-d231-4dc7-a08a-04110d047bd9\") "
Nov 02 14:30:04.457186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.457156    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-openshift-ca" pod "04c74a77-d231-4dc7-a08a-04110d047bd9" (UID: "04c74a77-d231-4dc7-a08a-04110d047bd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:04.458262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.458237    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:04.476581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.476540    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-xks-ca" pod "04c74a77-d231-4dc7-a08a-04110d047bd9" (UID: "04c74a77-d231-4dc7-a08a-04110d047bd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:30:04.478814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.478745    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c74a77-d231-4dc7-a08a-04110d047bd9-kube-api-access-sbn2x" pod "04c74a77-d231-4dc7-a08a-04110d047bd9" (UID: "04c74a77-d231-4dc7-a08a-04110d047bd9"). InnerVolumeSpecName "kube-api-access-sbn2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:04.559947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.559917    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04c74a77-d231-4dc7-a08a-04110d047bd9-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:04.560054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.559951    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbn2x\" (UniqueName: \"kubernetes.io/projected/04c74a77-d231-4dc7-a08a-04110d047bd9-kube-api-access-sbn2x\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:04.635733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.635692    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" event={"ID":"ce138676-ad2e-4761-ad72-571e0b3bf23c","Type":"ContainerStarted","Data":"6a15e7f98bdf23551a99b1ea88a81284cdaef5eaf2b128fa6908fc366297892a"}
Nov 02 14:30:04.637639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.637611    5034 generic.go:356] "Generic (PLEG): container finished" podID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerID="9ae6f90d9d4c0d17ac4e77125df5c28010dca2b5d2ea2c114393e3e15f3c8a46" exitCode=60
Nov 02 14:30:04.637836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.637741    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" event={"ID":"6d7b00a2-30e2-4c1f-bc50-d98158707200","Type":"ContainerDied","Data":"9ae6f90d9d4c0d17ac4e77125df5c28010dca2b5d2ea2c114393e3e15f3c8a46"}
Nov 02 14:30:04.638082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.638063    5034 scope.go:122] "RemoveContainer" containerID="9ae6f90d9d4c0d17ac4e77125df5c28010dca2b5d2ea2c114393e3e15f3c8a46"
Nov 02 14:30:04.640224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.640198    5034 generic.go:356] "Generic (PLEG): container finished" podID="798e455b-975d-468b-bc10-3e020e992626" containerID="3d0c265ae495fd801d8411c114f85497073bc5e501b33e96ea46251c732e8deb" exitCode=60
Nov 02 14:30:04.640330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.640267    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" event={"ID":"798e455b-975d-468b-bc10-3e020e992626","Type":"ContainerDied","Data":"3d0c265ae495fd801d8411c114f85497073bc5e501b33e96ea46251c732e8deb"}
Nov 02 14:30:04.640581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.640561    5034 scope.go:122] "RemoveContainer" containerID="3d0c265ae495fd801d8411c114f85497073bc5e501b33e96ea46251c732e8deb"
Nov 02 14:30:04.641708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.641685    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" event={"ID":"9a96c9cc-4506-402a-8923-620b9381ecc9","Type":"ContainerStarted","Data":"77ad31de9317bb8f723f0bb1c6e61974058dce5f4d8d62b0a1ac4236a68321bb"}
Nov 02 14:30:04.643078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.643060    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9" event={"ID":"04c74a77-d231-4dc7-a08a-04110d047bd9","Type":"ContainerDied","Data":"0abda4fb4b94c52518bf5f8a55f3f064bbe9ccea843d9243041145b5e3d5aceb"}
Nov 02 14:30:04.643158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.643095    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0abda4fb4b94c52518bf5f8a55f3f064bbe9ccea843d9243041145b5e3d5aceb"
Nov 02 14:30:04.643158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.643142    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"
Nov 02 14:30:04.645181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.645158    5034 generic.go:356] "Generic (PLEG): container finished" podID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerID="a4849396f227337577d57d8b4a7a8880fa23e63611f2ff2752d55283c59e3425" exitCode=60
Nov 02 14:30:04.645271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.645237    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" event={"ID":"00f54dd3-b21f-4705-af6f-d29c35b9368c","Type":"ContainerDied","Data":"a4849396f227337577d57d8b4a7a8880fa23e63611f2ff2752d55283c59e3425"}
Nov 02 14:30:04.645488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.645466    5034 scope.go:122] "RemoveContainer" containerID="a4849396f227337577d57d8b4a7a8880fa23e63611f2ff2752d55283c59e3425"
Nov 02 14:30:04.647276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.647253    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" event={"ID":"5267a8a7-bd52-4e1b-87eb-d6c4085e0769","Type":"ContainerStarted","Data":"92ad76014769e205998561aa216dcd8da9bd0c9a845ac2a267391d8ea7f9ba81"}
Nov 02 14:30:04.648991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.648967    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" event={"ID":"110008e0-3d52-4923-9188-98bafbfccf54","Type":"ContainerStarted","Data":"7c3bf6e84203783a7d2138ba4280b5e1fdf930b4178cf6074c6386b90ae52733"}
Nov 02 14:30:04.656513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.656485    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" event={"ID":"786d86b7-0109-4fe1-abce-3960750cc80d","Type":"ContainerStarted","Data":"640a0bad5cd70944f139aaca9b8065ef520994ea53857e5ff2fe4ed5afe5cecd"}
Nov 02 14:30:04.658805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.658388    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" event={"ID":"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd","Type":"ContainerStarted","Data":"d2218203fe008bef1fa8fca7ebd3141060d7b5fdc372a72cb99cc171c048a038"}
Nov 02 14:30:04.660382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.660352    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" event={"ID":"1c6427b3-906d-47bc-938f-c38f0c1b5b5a","Type":"ContainerStarted","Data":"fa0ab1dca0657a819bafffc9e36747fa921fba6d816a6fa439807e5695451ae0"}
Nov 02 14:30:04.663042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.663005    5034 generic.go:356] "Generic (PLEG): container finished" podID="12c10b34-f265-434c-a511-077efe1688d0" containerID="c01e9695159bf8ffd5da6d5298afc74e29ef51d2b156b2951f8b0dd0da4a6d71" exitCode=60
Nov 02 14:30:04.663153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.663088    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" event={"ID":"12c10b34-f265-434c-a511-077efe1688d0","Type":"ContainerDied","Data":"c01e9695159bf8ffd5da6d5298afc74e29ef51d2b156b2951f8b0dd0da4a6d71"}
Nov 02 14:30:04.663421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.663403    5034 scope.go:122] "RemoveContainer" containerID="c01e9695159bf8ffd5da6d5298afc74e29ef51d2b156b2951f8b0dd0da4a6d71"
Nov 02 14:30:04.665163 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.665142    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" event={"ID":"732c703d-dd22-4a0b-a2c0-93e2ac208979","Type":"ContainerStarted","Data":"4360ed860c9654956784e37dede48af7574639c7e57e5156ec277e047c31626e"}
Nov 02 14:30:04.666862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.666836    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" event={"ID":"1c667d91-1fd4-438c-b3a3-8281cf003b88","Type":"ContainerStarted","Data":"1b61f60e0cd00cf8f3d70583160d86da6c92ea32bc33861678ae682787bde3af"}
Nov 02 14:30:04.683235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.683178    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" podStartSLOduration=4.683160729 podStartE2EDuration="4.683160729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:04.682076587 +0000 UTC m=+2795.433381037" watchObservedRunningTime="2026-11-02 14:30:04.683160729 +0000 UTC m=+2795.434465167"
Nov 02 14:30:04.701326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.701262    5034 generic.go:356] "Generic (PLEG): container finished" podID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerID="f0aacb8a1efaa495b691f806dd19e9301845ed217316d070ccfdb67da8c40f88" exitCode=60
Nov 02 14:30:04.701431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.701411    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" event={"ID":"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68","Type":"ContainerDied","Data":"f0aacb8a1efaa495b691f806dd19e9301845ed217316d070ccfdb67da8c40f88"}
Nov 02 14:30:04.701974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.701948    5034 scope.go:122] "RemoveContainer" containerID="f0aacb8a1efaa495b691f806dd19e9301845ed217316d070ccfdb67da8c40f88"
Nov 02 14:30:04.710826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.710747    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" event={"ID":"5391b42a-6255-4531-957f-31dd8616f975","Type":"ContainerStarted","Data":"4ba183242cd63eced2ec93e2dd136bcd0642df123b0525680395ff667d2b6580"}
Nov 02 14:30:04.826086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.826012    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" podStartSLOduration=4.82598957 podStartE2EDuration="4.82598957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:04.819848503 +0000 UTC m=+2795.571152941" watchObservedRunningTime="2026-11-02 14:30:04.82598957 +0000 UTC m=+2795.577294008"
Nov 02 14:30:04.878721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.878620    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" podStartSLOduration=4.87860065 podStartE2EDuration="4.87860065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:04.866234556 +0000 UTC m=+2795.617538993" watchObservedRunningTime="2026-11-02 14:30:04.87860065 +0000 UTC m=+2795.629905088"
Nov 02 14:30:04.913818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:04.913742    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" podStartSLOduration=4.913727982 podStartE2EDuration="4.913727982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:04.910653724 +0000 UTC m=+2795.661958163" watchObservedRunningTime="2026-11-02 14:30:04.913727982 +0000 UTC m=+2795.665032420"
Nov 02 14:30:05.078817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.078536    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" podStartSLOduration=5.078518865 podStartE2EDuration="5.078518865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:05.076217437 +0000 UTC m=+2795.827521876" watchObservedRunningTime="2026-11-02 14:30:05.078518865 +0000 UTC m=+2795.829823304"
Nov 02 14:30:05.724322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.724287    5034 generic.go:356] "Generic (PLEG): container finished" podID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerID="6a15e7f98bdf23551a99b1ea88a81284cdaef5eaf2b128fa6908fc366297892a" exitCode=60
Nov 02 14:30:05.724760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.724366    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" event={"ID":"ce138676-ad2e-4761-ad72-571e0b3bf23c","Type":"ContainerDied","Data":"6a15e7f98bdf23551a99b1ea88a81284cdaef5eaf2b128fa6908fc366297892a"}
Nov 02 14:30:05.724760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.724739    5034 scope.go:122] "RemoveContainer" containerID="6a15e7f98bdf23551a99b1ea88a81284cdaef5eaf2b128fa6908fc366297892a"
Nov 02 14:30:05.726747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.726723    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" event={"ID":"6d7b00a2-30e2-4c1f-bc50-d98158707200","Type":"ContainerStarted","Data":"da88c65717ea701feb56a6cc7f7a0862165ca790a166172f3d0a0408787a7d00"}
Nov 02 14:30:05.729618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.729526    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" event={"ID":"798e455b-975d-468b-bc10-3e020e992626","Type":"ContainerStarted","Data":"ac021091e086b80016555e177ce22f9759825fa8b714d428f52c336c5cbde16d"}
Nov 02 14:30:05.731511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.731477    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" event={"ID":"9a96c9cc-4506-402a-8923-620b9381ecc9","Type":"ContainerStarted","Data":"b743b6c4a1dab5fa9e98d480b31ed9dc3d7fedf3c050e36dc8dd259a4cb9eba5"}
Nov 02 14:30:05.733698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.733674    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" event={"ID":"00f54dd3-b21f-4705-af6f-d29c35b9368c","Type":"ContainerStarted","Data":"e041a6f5e9e5bbfed029fd9b7cddda9daeace8936d7b30169612a12c9a4458b7"}
Nov 02 14:30:05.735567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.735547    5034 generic.go:356] "Generic (PLEG): container finished" podID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerID="92ad76014769e205998561aa216dcd8da9bd0c9a845ac2a267391d8ea7f9ba81" exitCode=60
Nov 02 14:30:05.735666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.735628    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" event={"ID":"5267a8a7-bd52-4e1b-87eb-d6c4085e0769","Type":"ContainerDied","Data":"92ad76014769e205998561aa216dcd8da9bd0c9a845ac2a267391d8ea7f9ba81"}
Nov 02 14:30:05.735878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.735860    5034 scope.go:122] "RemoveContainer" containerID="92ad76014769e205998561aa216dcd8da9bd0c9a845ac2a267391d8ea7f9ba81"
Nov 02 14:30:05.737237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.737214    5034 generic.go:356] "Generic (PLEG): container finished" podID="110008e0-3d52-4923-9188-98bafbfccf54" containerID="7c3bf6e84203783a7d2138ba4280b5e1fdf930b4178cf6074c6386b90ae52733" exitCode=60
Nov 02 14:30:05.737350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.737298    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" event={"ID":"110008e0-3d52-4923-9188-98bafbfccf54","Type":"ContainerDied","Data":"7c3bf6e84203783a7d2138ba4280b5e1fdf930b4178cf6074c6386b90ae52733"}
Nov 02 14:30:05.737489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.737476    5034 scope.go:122] "RemoveContainer" containerID="7c3bf6e84203783a7d2138ba4280b5e1fdf930b4178cf6074c6386b90ae52733"
Nov 02 14:30:05.738959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.738938    5034 generic.go:356] "Generic (PLEG): container finished" podID="786d86b7-0109-4fe1-abce-3960750cc80d" containerID="640a0bad5cd70944f139aaca9b8065ef520994ea53857e5ff2fe4ed5afe5cecd" exitCode=60
Nov 02 14:30:05.739061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.739004    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" event={"ID":"786d86b7-0109-4fe1-abce-3960750cc80d","Type":"ContainerDied","Data":"640a0bad5cd70944f139aaca9b8065ef520994ea53857e5ff2fe4ed5afe5cecd"}
Nov 02 14:30:05.739215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.739200    5034 scope.go:122] "RemoveContainer" containerID="640a0bad5cd70944f139aaca9b8065ef520994ea53857e5ff2fe4ed5afe5cecd"
Nov 02 14:30:05.740729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.740617    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" event={"ID":"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd","Type":"ContainerStarted","Data":"cac2429250ee23ae03cfdaf23c482e48cda29210075304787b0f49cf4ccdb85a"}
Nov 02 14:30:05.744715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.744686    5034 generic.go:356] "Generic (PLEG): container finished" podID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerID="fa0ab1dca0657a819bafffc9e36747fa921fba6d816a6fa439807e5695451ae0" exitCode=60
Nov 02 14:30:05.744900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.744844    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" event={"ID":"1c6427b3-906d-47bc-938f-c38f0c1b5b5a","Type":"ContainerDied","Data":"fa0ab1dca0657a819bafffc9e36747fa921fba6d816a6fa439807e5695451ae0"}
Nov 02 14:30:05.745050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.745032    5034 scope.go:122] "RemoveContainer" containerID="fa0ab1dca0657a819bafffc9e36747fa921fba6d816a6fa439807e5695451ae0"
Nov 02 14:30:05.747377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.747345    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" event={"ID":"12c10b34-f265-434c-a511-077efe1688d0","Type":"ContainerStarted","Data":"c7ae8aac0380e188698b7be3d8888966f53c538ee3f1e469aa92b12ab9729a14"}
Nov 02 14:30:05.749096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.749046    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" event={"ID":"732c703d-dd22-4a0b-a2c0-93e2ac208979","Type":"ContainerStarted","Data":"9c21c5247e2541b8f22e2cec4a2bbffcc0a4136b08a4124f30bcd48f67d83e3e"}
Nov 02 14:30:05.750844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.750823    5034 generic.go:356] "Generic (PLEG): container finished" podID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerID="28fd89dd8aaef4251a9ab5a0407bfc758727bee8af8f3c7337ac80221bdbb98a" exitCode=60
Nov 02 14:30:05.750950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.750899    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" event={"ID":"1c667d91-1fd4-438c-b3a3-8281cf003b88","Type":"ContainerDied","Data":"28fd89dd8aaef4251a9ab5a0407bfc758727bee8af8f3c7337ac80221bdbb98a"}
Nov 02 14:30:05.751149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.751125    5034 scope.go:122] "RemoveContainer" containerID="28fd89dd8aaef4251a9ab5a0407bfc758727bee8af8f3c7337ac80221bdbb98a"
Nov 02 14:30:05.752603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.752579    5034 generic.go:356] "Generic (PLEG): container finished" podID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerID="7f65dd064b15ef57836e37dd4d67d61195820accf20e3e5fbbb8244b5729c7cc" exitCode=60
Nov 02 14:30:05.752710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.752659    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" event={"ID":"a668bbd8-8a19-42f0-b635-7a88c4cd6b35","Type":"ContainerDied","Data":"7f65dd064b15ef57836e37dd4d67d61195820accf20e3e5fbbb8244b5729c7cc"}
Nov 02 14:30:05.752865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.752850    5034 scope.go:122] "RemoveContainer" containerID="7f65dd064b15ef57836e37dd4d67d61195820accf20e3e5fbbb8244b5729c7cc"
Nov 02 14:30:05.754905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.754883    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" event={"ID":"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68","Type":"ContainerStarted","Data":"e981499680e7e43459fbc9b608cca5a2edb2c632de258456c3c4c8ba86e451c8"}
Nov 02 14:30:05.757683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.757576    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" event={"ID":"5391b42a-6255-4531-957f-31dd8616f975","Type":"ContainerStarted","Data":"c91e4837076cd85f5b5ca8a8d039b1bf897c743c7d422064aaca2301ca5aa1ee"}
Nov 02 14:30:05.813389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.813332    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" podStartSLOduration=5.813318035 podStartE2EDuration="5.813318035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:05.810850308 +0000 UTC m=+2796.562154772" watchObservedRunningTime="2026-11-02 14:30:05.813318035 +0000 UTC m=+2796.564622472"
Nov 02 14:30:05.861284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.861226    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" podStartSLOduration=5.861211531 podStartE2EDuration="5.861211531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:05.85534934 +0000 UTC m=+2796.606653780" watchObservedRunningTime="2026-11-02 14:30:05.861211531 +0000 UTC m=+2796.612515968"
Nov 02 14:30:05.904374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:05.904320    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" podStartSLOduration=5.904307001 podStartE2EDuration="5.904307001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:05.901829673 +0000 UTC m=+2796.653134112" watchObservedRunningTime="2026-11-02 14:30:05.904307001 +0000 UTC m=+2796.655611478"
Nov 02 14:30:06.041537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.041488    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" podStartSLOduration=6.041474465 podStartE2EDuration="6.041474465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:06.034885178 +0000 UTC m=+2796.786189616" watchObservedRunningTime="2026-11-02 14:30:06.041474465 +0000 UTC m=+2796.792778903"
Nov 02 14:30:06.083158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.083096    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" podStartSLOduration=6.083074324 podStartE2EDuration="6.083074324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:06.078490592 +0000 UTC m=+2796.829795032" watchObservedRunningTime="2026-11-02 14:30:06.083074324 +0000 UTC m=+2796.834378767"
Nov 02 14:30:06.133396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.133337    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" podStartSLOduration=6.133322223 podStartE2EDuration="6.133322223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:06.128066906 +0000 UTC m=+2796.879371347" watchObservedRunningTime="2026-11-02 14:30:06.133322223 +0000 UTC m=+2796.884626661"
Nov 02 14:30:06.388889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.388838    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" podStartSLOduration=6.388822168 podStartE2EDuration="6.388822168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:06.388706521 +0000 UTC m=+2797.140010961" watchObservedRunningTime="2026-11-02 14:30:06.388822168 +0000 UTC m=+2797.140126608"
Nov 02 14:30:06.444005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.443944    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" podStartSLOduration=6.443928279 podStartE2EDuration="6.443928279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:06.437944316 +0000 UTC m=+2797.189248797" watchObservedRunningTime="2026-11-02 14:30:06.443928279 +0000 UTC m=+2797.195232717"
Nov 02 14:30:06.770092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.769233    5034 generic.go:356] "Generic (PLEG): container finished" podID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerID="e981499680e7e43459fbc9b608cca5a2edb2c632de258456c3c4c8ba86e451c8" exitCode=60
Nov 02 14:30:06.770092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.769384    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" event={"ID":"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68","Type":"ContainerDied","Data":"e981499680e7e43459fbc9b608cca5a2edb2c632de258456c3c4c8ba86e451c8"}
Nov 02 14:30:06.770092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.769456    5034 scope.go:122] "RemoveContainer" containerID="f0aacb8a1efaa495b691f806dd19e9301845ed217316d070ccfdb67da8c40f88"
Nov 02 14:30:06.770980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.770177    5034 scope.go:122] "RemoveContainer" containerID="e981499680e7e43459fbc9b608cca5a2edb2c632de258456c3c4c8ba86e451c8"
Nov 02 14:30:06.770980 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:06.770702    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh_odh-ai-gateway-infra(d1bf66ee-0739-4042-9f1d-d1f0f5de7f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68"
Nov 02 14:30:06.772864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.772730    5034 generic.go:356] "Generic (PLEG): container finished" podID="5391b42a-6255-4531-957f-31dd8616f975" containerID="c91e4837076cd85f5b5ca8a8d039b1bf897c743c7d422064aaca2301ca5aa1ee" exitCode=60
Nov 02 14:30:06.772982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.772892    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" event={"ID":"5391b42a-6255-4531-957f-31dd8616f975","Type":"ContainerDied","Data":"c91e4837076cd85f5b5ca8a8d039b1bf897c743c7d422064aaca2301ca5aa1ee"}
Nov 02 14:30:06.773412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.773391    5034 scope.go:122] "RemoveContainer" containerID="c91e4837076cd85f5b5ca8a8d039b1bf897c743c7d422064aaca2301ca5aa1ee"
Nov 02 14:30:06.778904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.778720    5034 generic.go:356] "Generic (PLEG): container finished" podID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerID="da88c65717ea701feb56a6cc7f7a0862165ca790a166172f3d0a0408787a7d00" exitCode=60
Nov 02 14:30:06.778982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.778893    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" event={"ID":"6d7b00a2-30e2-4c1f-bc50-d98158707200","Type":"ContainerDied","Data":"da88c65717ea701feb56a6cc7f7a0862165ca790a166172f3d0a0408787a7d00"}
Nov 02 14:30:06.779212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.779190    5034 scope.go:122] "RemoveContainer" containerID="da88c65717ea701feb56a6cc7f7a0862165ca790a166172f3d0a0408787a7d00"
Nov 02 14:30:06.779670 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:06.779598    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7_odh-ai-gateway-infra(6d7b00a2-30e2-4c1f-bc50-d98158707200)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200"
Nov 02 14:30:06.783046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.782500    5034 generic.go:356] "Generic (PLEG): container finished" podID="798e455b-975d-468b-bc10-3e020e992626" containerID="ac021091e086b80016555e177ce22f9759825fa8b714d428f52c336c5cbde16d" exitCode=60
Nov 02 14:30:06.783046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.782580    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" event={"ID":"798e455b-975d-468b-bc10-3e020e992626","Type":"ContainerDied","Data":"ac021091e086b80016555e177ce22f9759825fa8b714d428f52c336c5cbde16d"}
Nov 02 14:30:06.783209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.783069    5034 scope.go:122] "RemoveContainer" containerID="ac021091e086b80016555e177ce22f9759825fa8b714d428f52c336c5cbde16d"
Nov 02 14:30:06.783506 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:06.783449    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl_odh-ai-gateway-infra(798e455b-975d-468b-bc10-3e020e992626)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" podUID="798e455b-975d-468b-bc10-3e020e992626"
Nov 02 14:30:06.785068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.785047    5034 generic.go:356] "Generic (PLEG): container finished" podID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerID="b743b6c4a1dab5fa9e98d480b31ed9dc3d7fedf3c050e36dc8dd259a4cb9eba5" exitCode=60
Nov 02 14:30:06.785185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.785155    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" event={"ID":"9a96c9cc-4506-402a-8923-620b9381ecc9","Type":"ContainerDied","Data":"b743b6c4a1dab5fa9e98d480b31ed9dc3d7fedf3c050e36dc8dd259a4cb9eba5"}
Nov 02 14:30:06.785535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.785517    5034 scope.go:122] "RemoveContainer" containerID="b743b6c4a1dab5fa9e98d480b31ed9dc3d7fedf3c050e36dc8dd259a4cb9eba5"
Nov 02 14:30:06.788195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.788173    5034 generic.go:356] "Generic (PLEG): container finished" podID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerID="e041a6f5e9e5bbfed029fd9b7cddda9daeace8936d7b30169612a12c9a4458b7" exitCode=60
Nov 02 14:30:06.788310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.788200    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" event={"ID":"00f54dd3-b21f-4705-af6f-d29c35b9368c","Type":"ContainerDied","Data":"e041a6f5e9e5bbfed029fd9b7cddda9daeace8936d7b30169612a12c9a4458b7"}
Nov 02 14:30:06.788566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.788548    5034 scope.go:122] "RemoveContainer" containerID="e041a6f5e9e5bbfed029fd9b7cddda9daeace8936d7b30169612a12c9a4458b7"
Nov 02 14:30:06.788995 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:06.788971    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48_odh-ai-gateway-infra(00f54dd3-b21f-4705-af6f-d29c35b9368c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c"
Nov 02 14:30:06.795522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.795494    5034 generic.go:356] "Generic (PLEG): container finished" podID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerID="cac2429250ee23ae03cfdaf23c482e48cda29210075304787b0f49cf4ccdb85a" exitCode=60
Nov 02 14:30:06.795672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.795609    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" event={"ID":"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd","Type":"ContainerDied","Data":"cac2429250ee23ae03cfdaf23c482e48cda29210075304787b0f49cf4ccdb85a"}
Nov 02 14:30:06.796259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.796235    5034 scope.go:122] "RemoveContainer" containerID="cac2429250ee23ae03cfdaf23c482e48cda29210075304787b0f49cf4ccdb85a"
Nov 02 14:30:06.800136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.800103    5034 generic.go:356] "Generic (PLEG): container finished" podID="12c10b34-f265-434c-a511-077efe1688d0" containerID="c7ae8aac0380e188698b7be3d8888966f53c538ee3f1e469aa92b12ab9729a14" exitCode=60
Nov 02 14:30:06.800249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.800188    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" event={"ID":"12c10b34-f265-434c-a511-077efe1688d0","Type":"ContainerDied","Data":"c7ae8aac0380e188698b7be3d8888966f53c538ee3f1e469aa92b12ab9729a14"}
Nov 02 14:30:06.800505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.800486    5034 scope.go:122] "RemoveContainer" containerID="c7ae8aac0380e188698b7be3d8888966f53c538ee3f1e469aa92b12ab9729a14"
Nov 02 14:30:06.800877 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:06.800850    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb_odh-ai-gateway-infra(12c10b34-f265-434c-a511-077efe1688d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" podUID="12c10b34-f265-434c-a511-077efe1688d0"
Nov 02 14:30:06.801868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.801843    5034 generic.go:356] "Generic (PLEG): container finished" podID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerID="9c21c5247e2541b8f22e2cec4a2bbffcc0a4136b08a4124f30bcd48f67d83e3e" exitCode=60
Nov 02 14:30:06.801952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.801895    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" event={"ID":"732c703d-dd22-4a0b-a2c0-93e2ac208979","Type":"ContainerDied","Data":"9c21c5247e2541b8f22e2cec4a2bbffcc0a4136b08a4124f30bcd48f67d83e3e"}
Nov 02 14:30:06.802220 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.802201    5034 scope.go:122] "RemoveContainer" containerID="9c21c5247e2541b8f22e2cec4a2bbffcc0a4136b08a4124f30bcd48f67d83e3e"
Nov 02 14:30:06.806255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:06.806235    5034 scope.go:122] "RemoveContainer" containerID="9ae6f90d9d4c0d17ac4e77125df5c28010dca2b5d2ea2c114393e3e15f3c8a46"
Nov 02 14:30:07.119188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.119158    5034 scope.go:122] "RemoveContainer" containerID="3d0c265ae495fd801d8411c114f85497073bc5e501b33e96ea46251c732e8deb"
Nov 02 14:30:07.130827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.130578    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"]
Nov 02 14:30:07.132131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.132070    5034 cpu_manager.go:414] "containerMap: removing container" podUID="04c74a77-d231-4dc7-a08a-04110d047bd9" containerName="revoke-keys"
Nov 02 14:30:07.132131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.132096    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04c74a77-d231-4dc7-a08a-04110d047bd9" containerName="revoke-keys"
Nov 02 14:30:07.134647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.132423    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="04c74a77-d231-4dc7-a08a-04110d047bd9" containerName="revoke-keys"
Nov 02 14:30:07.139239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.139215    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.157592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.157552    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"]
Nov 02 14:30:07.187377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.187216    5034 scope.go:122] "RemoveContainer" containerID="a4849396f227337577d57d8b4a7a8880fa23e63611f2ff2752d55283c59e3425"
Nov 02 14:30:07.209524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.209505    5034 scope.go:122] "RemoveContainer" containerID="c01e9695159bf8ffd5da6d5298afc74e29ef51d2b156b2951f8b0dd0da4a6d71"
Nov 02 14:30:07.236452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.236417    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.236616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.236588    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.236720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.236698    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.236768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.236739    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.236914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.236895    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vsn\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-kube-api-access-j9vsn\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.236963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.236933    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.237081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.237065    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.237122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.237113    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.237229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.237185    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b44b24-e712-4d2e-9a1d-9c219925ff44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338265    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338325    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338362    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338400    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vsn\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-kube-api-access-j9vsn\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338435    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338544    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338608    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338716    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b44b24-e712-4d2e-9a1d-9c219925ff44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338794    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.338936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.338863    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.339321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.339118    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.339428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.339377    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.339642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.339609    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.339837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.339815    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b44b24-e712-4d2e-9a1d-9c219925ff44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.373392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.373312    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.373588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.373531    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.374080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.374054    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.375630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.375577    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vsn\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-kube-api-access-j9vsn\") pod \"maas-default-gateway-openshift-default-6886b96f9-qkbsm\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.506870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.506838    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:07.814772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.814728    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" event={"ID":"5391b42a-6255-4531-957f-31dd8616f975","Type":"ContainerStarted","Data":"b99e0de451512811d3cb8b8cfd7e54fe15ad6901af49dd0f9468dde948c089db"}
Nov 02 14:30:07.816930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.816900    5034 generic.go:356] "Generic (PLEG): container finished" podID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerID="1db6c8844fcecca73dea977079c95c2dffa2e090f23d39fc50265fbe3227a530" exitCode=60
Nov 02 14:30:07.817072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.816980    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" event={"ID":"ce138676-ad2e-4761-ad72-571e0b3bf23c","Type":"ContainerDied","Data":"1db6c8844fcecca73dea977079c95c2dffa2e090f23d39fc50265fbe3227a530"}
Nov 02 14:30:07.817072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.817036    5034 scope.go:122] "RemoveContainer" containerID="6a15e7f98bdf23551a99b1ea88a81284cdaef5eaf2b128fa6908fc366297892a"
Nov 02 14:30:07.817336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.817301    5034 scope.go:122] "RemoveContainer" containerID="1db6c8844fcecca73dea977079c95c2dffa2e090f23d39fc50265fbe3227a530"
Nov 02 14:30:07.817716 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.817690    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw_odh-ai-gateway-infra(ce138676-ad2e-4761-ad72-571e0b3bf23c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c"
Nov 02 14:30:07.819328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.819307    5034 scope.go:122] "RemoveContainer" containerID="da88c65717ea701feb56a6cc7f7a0862165ca790a166172f3d0a0408787a7d00"
Nov 02 14:30:07.819699 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.819637    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7_odh-ai-gateway-infra(6d7b00a2-30e2-4c1f-bc50-d98158707200)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200"
Nov 02 14:30:07.821192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.821172    5034 scope.go:122] "RemoveContainer" containerID="ac021091e086b80016555e177ce22f9759825fa8b714d428f52c336c5cbde16d"
Nov 02 14:30:07.821518 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.821490    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl_odh-ai-gateway-infra(798e455b-975d-468b-bc10-3e020e992626)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" podUID="798e455b-975d-468b-bc10-3e020e992626"
Nov 02 14:30:07.823222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.823206    5034 scope.go:122] "RemoveContainer" containerID="e041a6f5e9e5bbfed029fd9b7cddda9daeace8936d7b30169612a12c9a4458b7"
Nov 02 14:30:07.823518 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.823493    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48_odh-ai-gateway-infra(00f54dd3-b21f-4705-af6f-d29c35b9368c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c"
Nov 02 14:30:07.825142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.825117    5034 generic.go:356] "Generic (PLEG): container finished" podID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerID="416b0020b6c1074a0d0bbaad152311e041a3061fd1f6798eeec40f9b23008b30" exitCode=60
Nov 02 14:30:07.825231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.825200    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" event={"ID":"5267a8a7-bd52-4e1b-87eb-d6c4085e0769","Type":"ContainerDied","Data":"416b0020b6c1074a0d0bbaad152311e041a3061fd1f6798eeec40f9b23008b30"}
Nov 02 14:30:07.825451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.825434    5034 scope.go:122] "RemoveContainer" containerID="416b0020b6c1074a0d0bbaad152311e041a3061fd1f6798eeec40f9b23008b30"
Nov 02 14:30:07.825704 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.825688    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb_odh-ai-gateway-infra(5267a8a7-bd52-4e1b-87eb-d6c4085e0769)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769"
Nov 02 14:30:07.827572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.827552    5034 generic.go:356] "Generic (PLEG): container finished" podID="110008e0-3d52-4923-9188-98bafbfccf54" containerID="3839801e3c67917ee747645cf88d334ce9f8448c259b037f363e794e3f9740bf" exitCode=60
Nov 02 14:30:07.827673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.827576    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" event={"ID":"110008e0-3d52-4923-9188-98bafbfccf54","Type":"ContainerDied","Data":"3839801e3c67917ee747645cf88d334ce9f8448c259b037f363e794e3f9740bf"}
Nov 02 14:30:07.827989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.827928    5034 scope.go:122] "RemoveContainer" containerID="3839801e3c67917ee747645cf88d334ce9f8448c259b037f363e794e3f9740bf"
Nov 02 14:30:07.828233 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.828212    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj_odh-ai-gateway-infra(110008e0-3d52-4923-9188-98bafbfccf54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" podUID="110008e0-3d52-4923-9188-98bafbfccf54"
Nov 02 14:30:07.829758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.829727    5034 generic.go:356] "Generic (PLEG): container finished" podID="786d86b7-0109-4fe1-abce-3960750cc80d" containerID="759aff80ff571981c038a04af584cd3d57b2aad37839c1c4e433fce468b94688" exitCode=60
Nov 02 14:30:07.829879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.829815    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" event={"ID":"786d86b7-0109-4fe1-abce-3960750cc80d","Type":"ContainerDied","Data":"759aff80ff571981c038a04af584cd3d57b2aad37839c1c4e433fce468b94688"}
Nov 02 14:30:07.830161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.830144    5034 scope.go:122] "RemoveContainer" containerID="759aff80ff571981c038a04af584cd3d57b2aad37839c1c4e433fce468b94688"
Nov 02 14:30:07.830445 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.830421    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6_odh-ai-gateway-infra(786d86b7-0109-4fe1-abce-3960750cc80d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" podUID="786d86b7-0109-4fe1-abce-3960750cc80d"
Nov 02 14:30:07.832305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.832279    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" event={"ID":"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd","Type":"ContainerStarted","Data":"a4b5bedbc930eac645478b2f9463077508bce65ee67b18240933f519baedb209"}
Nov 02 14:30:07.835735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.835709    5034 generic.go:356] "Generic (PLEG): container finished" podID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerID="bc56e824ae5d33698461bbe334d9fdbdd3d3d3ac4ad4b0c8b1302c2061751726" exitCode=60
Nov 02 14:30:07.835864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.835796    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" event={"ID":"1c6427b3-906d-47bc-938f-c38f0c1b5b5a","Type":"ContainerDied","Data":"bc56e824ae5d33698461bbe334d9fdbdd3d3d3ac4ad4b0c8b1302c2061751726"}
Nov 02 14:30:07.837063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.837037    5034 scope.go:122] "RemoveContainer" containerID="bc56e824ae5d33698461bbe334d9fdbdd3d3d3ac4ad4b0c8b1302c2061751726"
Nov 02 14:30:07.837390 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.837367    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd_odh-ai-gateway-infra(1c6427b3-906d-47bc-938f-c38f0c1b5b5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a"
Nov 02 14:30:07.838072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.838053    5034 scope.go:122] "RemoveContainer" containerID="c7ae8aac0380e188698b7be3d8888966f53c538ee3f1e469aa92b12ab9729a14"
Nov 02 14:30:07.838342 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.838320    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb_odh-ai-gateway-infra(12c10b34-f265-434c-a511-077efe1688d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" podUID="12c10b34-f265-434c-a511-077efe1688d0"
Nov 02 14:30:07.839810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.839790    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" event={"ID":"732c703d-dd22-4a0b-a2c0-93e2ac208979","Type":"ContainerStarted","Data":"c1538081a0a64c0264147e90bee1be2d56972d6ccba4608276f733a20114db68"}
Nov 02 14:30:07.841630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.841609    5034 generic.go:356] "Generic (PLEG): container finished" podID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerID="22c1b8c3c306dd3bd0e27f76cf755fcaaa71285cf48eed7e8bf8ea9ff2b97e6c" exitCode=60
Nov 02 14:30:07.841721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.841694    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" event={"ID":"1c667d91-1fd4-438c-b3a3-8281cf003b88","Type":"ContainerDied","Data":"22c1b8c3c306dd3bd0e27f76cf755fcaaa71285cf48eed7e8bf8ea9ff2b97e6c"}
Nov 02 14:30:07.841967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.841950    5034 scope.go:122] "RemoveContainer" containerID="22c1b8c3c306dd3bd0e27f76cf755fcaaa71285cf48eed7e8bf8ea9ff2b97e6c"
Nov 02 14:30:07.842226 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.842204    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg_odh-ai-gateway-infra(1c667d91-1fd4-438c-b3a3-8281cf003b88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88"
Nov 02 14:30:07.843626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.843604    5034 generic.go:356] "Generic (PLEG): container finished" podID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerID="8c161a6bda49a3254c941906822afb21d1a889742a56086537a06ae84603a288" exitCode=60
Nov 02 14:30:07.843719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.843626    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" event={"ID":"a668bbd8-8a19-42f0-b635-7a88c4cd6b35","Type":"ContainerDied","Data":"8c161a6bda49a3254c941906822afb21d1a889742a56086537a06ae84603a288"}
Nov 02 14:30:07.843905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.843891    5034 scope.go:122] "RemoveContainer" containerID="8c161a6bda49a3254c941906822afb21d1a889742a56086537a06ae84603a288"
Nov 02 14:30:07.844148 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.844128    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c_odh-ai-gateway-infra(a668bbd8-8a19-42f0-b635-7a88c4cd6b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35"
Nov 02 14:30:07.845526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.845511    5034 scope.go:122] "RemoveContainer" containerID="e981499680e7e43459fbc9b608cca5a2edb2c632de258456c3c4c8ba86e451c8"
Nov 02 14:30:07.845720 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:07.845703    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh_odh-ai-gateway-infra(d1bf66ee-0739-4042-9f1d-d1f0f5de7f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68"
Nov 02 14:30:07.911467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.911438    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"]
Nov 02 14:30:07.956942 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:07.956888    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b44b24_e712_4d2e_9a1d_9c219925ff44.slice/crio-5f56659685e4b2106a4358961ccd7e158bda11d205fc956ee1e867b6efb36dc9 WatchSource:0}: Error finding container 5f56659685e4b2106a4358961ccd7e158bda11d205fc956ee1e867b6efb36dc9: Status 404 returned error can't find the container with id 5f56659685e4b2106a4358961ccd7e158bda11d205fc956ee1e867b6efb36dc9
Nov 02 14:30:07.965355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.965319    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:30:07.965493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.965387    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:30:07.965493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:07.965415    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:30:08.028583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.028557    5034 scope.go:122] "RemoveContainer" containerID="92ad76014769e205998561aa216dcd8da9bd0c9a845ac2a267391d8ea7f9ba81"
Nov 02 14:30:08.049999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.049956    5034 scope.go:122] "RemoveContainer" containerID="7c3bf6e84203783a7d2138ba4280b5e1fdf930b4178cf6074c6386b90ae52733"
Nov 02 14:30:08.074598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.073910    5034 scope.go:122] "RemoveContainer" containerID="640a0bad5cd70944f139aaca9b8065ef520994ea53857e5ff2fe4ed5afe5cecd"
Nov 02 14:30:08.129374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.129344    5034 scope.go:122] "RemoveContainer" containerID="fa0ab1dca0657a819bafffc9e36747fa921fba6d816a6fa439807e5695451ae0"
Nov 02 14:30:08.149413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.149386    5034 scope.go:122] "RemoveContainer" containerID="28fd89dd8aaef4251a9ab5a0407bfc758727bee8af8f3c7337ac80221bdbb98a"
Nov 02 14:30:08.168665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.168639    5034 scope.go:122] "RemoveContainer" containerID="7f65dd064b15ef57836e37dd4d67d61195820accf20e3e5fbbb8244b5729c7cc"
Nov 02 14:30:08.874541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.874512    5034 scope.go:122] "RemoveContainer" containerID="bc56e824ae5d33698461bbe334d9fdbdd3d3d3ac4ad4b0c8b1302c2061751726"
Nov 02 14:30:08.885468 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.885404    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd_odh-ai-gateway-infra(1c6427b3-906d-47bc-938f-c38f0c1b5b5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a"
Nov 02 14:30:08.905755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.905294    5034 scope.go:122] "RemoveContainer" containerID="c1538081a0a64c0264147e90bee1be2d56972d6ccba4608276f733a20114db68"
Nov 02 14:30:08.905755 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.905690    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9_odh-ai-gateway-infra(732c703d-dd22-4a0b-a2c0-93e2ac208979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979"
Nov 02 14:30:08.906225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.906043    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" event={"ID":"732c703d-dd22-4a0b-a2c0-93e2ac208979","Type":"ContainerDied","Data":"c1538081a0a64c0264147e90bee1be2d56972d6ccba4608276f733a20114db68"}
Nov 02 14:30:08.906225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.906142    5034 scope.go:122] "RemoveContainer" containerID="9c21c5247e2541b8f22e2cec4a2bbffcc0a4136b08a4124f30bcd48f67d83e3e"
Nov 02 14:30:08.907949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.907919    5034 generic.go:356] "Generic (PLEG): container finished" podID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerID="c1538081a0a64c0264147e90bee1be2d56972d6ccba4608276f733a20114db68" exitCode=60
Nov 02 14:30:08.912362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.911530    5034 scope.go:122] "RemoveContainer" containerID="22c1b8c3c306dd3bd0e27f76cf755fcaaa71285cf48eed7e8bf8ea9ff2b97e6c"
Nov 02 14:30:08.912362 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.911880    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg_odh-ai-gateway-infra(1c667d91-1fd4-438c-b3a3-8281cf003b88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88"
Nov 02 14:30:08.914912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.914469    5034 scope.go:122] "RemoveContainer" containerID="8c161a6bda49a3254c941906822afb21d1a889742a56086537a06ae84603a288"
Nov 02 14:30:08.914912 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.914869    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c_odh-ai-gateway-infra(a668bbd8-8a19-42f0-b635-7a88c4cd6b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35"
Nov 02 14:30:08.916175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.916152    5034 generic.go:356] "Generic (PLEG): container finished" podID="5391b42a-6255-4531-957f-31dd8616f975" containerID="b99e0de451512811d3cb8b8cfd7e54fe15ad6901af49dd0f9468dde948c089db" exitCode=60
Nov 02 14:30:08.916265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.916234    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" event={"ID":"5391b42a-6255-4531-957f-31dd8616f975","Type":"ContainerDied","Data":"b99e0de451512811d3cb8b8cfd7e54fe15ad6901af49dd0f9468dde948c089db"}
Nov 02 14:30:08.916558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.916543    5034 scope.go:122] "RemoveContainer" containerID="b99e0de451512811d3cb8b8cfd7e54fe15ad6901af49dd0f9468dde948c089db"
Nov 02 14:30:08.916847 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.916829    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2_odh-ai-gateway-infra(5391b42a-6255-4531-957f-31dd8616f975)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" podUID="5391b42a-6255-4531-957f-31dd8616f975"
Nov 02 14:30:08.918836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.918818    5034 scope.go:122] "RemoveContainer" containerID="1db6c8844fcecca73dea977079c95c2dffa2e090f23d39fc50265fbe3227a530"
Nov 02 14:30:08.919335 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.919311    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw_odh-ai-gateway-infra(ce138676-ad2e-4761-ad72-571e0b3bf23c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c"
Nov 02 14:30:08.920818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.920353    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" event={"ID":"28b44b24-e712-4d2e-9a1d-9c219925ff44","Type":"ContainerStarted","Data":"6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2"}
Nov 02 14:30:08.920818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.920399    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" event={"ID":"28b44b24-e712-4d2e-9a1d-9c219925ff44","Type":"ContainerStarted","Data":"5f56659685e4b2106a4358961ccd7e158bda11d205fc956ee1e867b6efb36dc9"}
Nov 02 14:30:08.923076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.923051    5034 generic.go:356] "Generic (PLEG): container finished" podID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerID="026b3728a2cf6f7d752de912d9f60c5f606da9bbacd0ff104738a11cc4624db9" exitCode=60
Nov 02 14:30:08.923212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.923196    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" event={"ID":"9a96c9cc-4506-402a-8923-620b9381ecc9","Type":"ContainerDied","Data":"026b3728a2cf6f7d752de912d9f60c5f606da9bbacd0ff104738a11cc4624db9"}
Nov 02 14:30:08.923597 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.923580    5034 scope.go:122] "RemoveContainer" containerID="026b3728a2cf6f7d752de912d9f60c5f606da9bbacd0ff104738a11cc4624db9"
Nov 02 14:30:08.923981 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.923955    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg_odh-ai-gateway-infra(9a96c9cc-4506-402a-8923-620b9381ecc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9"
Nov 02 14:30:08.925538 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.925520    5034 scope.go:122] "RemoveContainer" containerID="c91e4837076cd85f5b5ca8a8d039b1bf897c743c7d422064aaca2301ca5aa1ee"
Nov 02 14:30:08.925850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.925834    5034 scope.go:122] "RemoveContainer" containerID="416b0020b6c1074a0d0bbaad152311e041a3061fd1f6798eeec40f9b23008b30"
Nov 02 14:30:08.926107 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.926090    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb_odh-ai-gateway-infra(5267a8a7-bd52-4e1b-87eb-d6c4085e0769)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769"
Nov 02 14:30:08.927594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.927575    5034 scope.go:122] "RemoveContainer" containerID="3839801e3c67917ee747645cf88d334ce9f8448c259b037f363e794e3f9740bf"
Nov 02 14:30:08.927900 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.927877    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj_odh-ai-gateway-infra(110008e0-3d52-4923-9188-98bafbfccf54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" podUID="110008e0-3d52-4923-9188-98bafbfccf54"
Nov 02 14:30:08.930622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.930601    5034 scope.go:122] "RemoveContainer" containerID="759aff80ff571981c038a04af584cd3d57b2aad37839c1c4e433fce468b94688"
Nov 02 14:30:08.931271 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.931135    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6_odh-ai-gateway-infra(786d86b7-0109-4fe1-abce-3960750cc80d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" podUID="786d86b7-0109-4fe1-abce-3960750cc80d"
Nov 02 14:30:08.939488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.937750    5034 generic.go:356] "Generic (PLEG): container finished" podID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerID="a4b5bedbc930eac645478b2f9463077508bce65ee67b18240933f519baedb209" exitCode=60
Nov 02 14:30:08.939488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.937819    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" event={"ID":"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd","Type":"ContainerDied","Data":"a4b5bedbc930eac645478b2f9463077508bce65ee67b18240933f519baedb209"}
Nov 02 14:30:08.939488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.938201    5034 scope.go:122] "RemoveContainer" containerID="a4b5bedbc930eac645478b2f9463077508bce65ee67b18240933f519baedb209"
Nov 02 14:30:08.939488 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:08.938590    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4_odh-ai-gateway-infra(ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd"
Nov 02 14:30:08.945510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.945491    5034 scope.go:122] "RemoveContainer" containerID="b743b6c4a1dab5fa9e98d480b31ed9dc3d7fedf3c050e36dc8dd259a4cb9eba5"
Nov 02 14:30:08.967974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:08.964505    5034 scope.go:122] "RemoveContainer" containerID="cac2429250ee23ae03cfdaf23c482e48cda29210075304787b0f49cf4ccdb85a"
Nov 02 14:30:09.034268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:09.034206    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podStartSLOduration=2.034190679 podStartE2EDuration="2.034190679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:09.032726653 +0000 UTC m=+2799.784031093" watchObservedRunningTime="2026-11-02 14:30:09.034190679 +0000 UTC m=+2799.785495117"
Nov 02 14:30:09.507705 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:09.507660    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:09.510151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:09.510121    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qkbsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 14:30:09.510321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:09.510182    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 14:30:09.950427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:09.950294    5034 scope.go:122] "RemoveContainer" containerID="c1538081a0a64c0264147e90bee1be2d56972d6ccba4608276f733a20114db68"
Nov 02 14:30:09.950992 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:09.950656    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9_odh-ai-gateway-infra(732c703d-dd22-4a0b-a2c0-93e2ac208979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979"
Nov 02 14:30:09.952568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:09.952528    5034 scope.go:122] "RemoveContainer" containerID="b99e0de451512811d3cb8b8cfd7e54fe15ad6901af49dd0f9468dde948c089db"
Nov 02 14:30:09.953376 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:09.953043    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2_odh-ai-gateway-infra(5391b42a-6255-4531-957f-31dd8616f975)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" podUID="5391b42a-6255-4531-957f-31dd8616f975"
Nov 02 14:30:09.954820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:09.954800    5034 scope.go:122] "RemoveContainer" containerID="026b3728a2cf6f7d752de912d9f60c5f606da9bbacd0ff104738a11cc4624db9"
Nov 02 14:30:09.955279 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:09.955257    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg_odh-ai-gateway-infra(9a96c9cc-4506-402a-8923-620b9381ecc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9"
Nov 02 14:30:09.957374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:09.957358    5034 scope.go:122] "RemoveContainer" containerID="a4b5bedbc930eac645478b2f9463077508bce65ee67b18240933f519baedb209"
Nov 02 14:30:09.957650 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:09.957633    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4_odh-ai-gateway-infra(ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd"
Nov 02 14:30:10.511207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:10.510634    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qkbsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 14:30:10.511207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:10.510747    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 14:30:11.579816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.579743    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"]
Nov 02 14:30:11.608814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.603028    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qkbsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 14:30:11.610876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.610830    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 14:30:11.615071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.614007    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"]
Nov 02 14:30:11.615071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.614123    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:11.634863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.634826    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:30:11.818043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.818002    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-394ada8c-zdjgn\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:11.818492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.818462    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8zgn\" (UniqueName: \"kubernetes.io/projected/f451c332-f287-45c4-8286-9844268d9bd6-kube-api-access-c8zgn\") pod \"maas-api-revoke-sub-394ada8c-zdjgn\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:11.818842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.818818    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-394ada8c-zdjgn\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:11.934483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.934397    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-394ada8c-zdjgn\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:11.934758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.934740    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zgn\" (UniqueName: \"kubernetes.io/projected/f451c332-f287-45c4-8286-9844268d9bd6-kube-api-access-c8zgn\") pod \"maas-api-revoke-sub-394ada8c-zdjgn\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:11.935031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.935001    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-394ada8c-zdjgn\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:11.935631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.935606    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-394ada8c-zdjgn\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:11.944009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.937348    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-394ada8c-zdjgn\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:11.962106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:11.961551    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zgn\" (UniqueName: \"kubernetes.io/projected/f451c332-f287-45c4-8286-9844268d9bd6-kube-api-access-c8zgn\") pod \"maas-api-revoke-sub-394ada8c-zdjgn\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:12.250880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:12.248301    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:12.535898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:12.535075    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qkbsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 14:30:12.535898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:12.535131    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 14:30:12.712506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:12.712475    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"]
Nov 02 14:30:12.715081 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:12.715052    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf451c332_f287_45c4_8286_9844268d9bd6.slice/crio-3f0daca76136e111bcd093c8392ee1baef207a5d78065d45a2b465f4f1a48332 WatchSource:0}: Error finding container 3f0daca76136e111bcd093c8392ee1baef207a5d78065d45a2b465f4f1a48332: Status 404 returned error can't find the container with id 3f0daca76136e111bcd093c8392ee1baef207a5d78065d45a2b465f4f1a48332
Nov 02 14:30:13.090640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:13.090585    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn" event={"ID":"f451c332-f287-45c4-8286-9844268d9bd6","Type":"ContainerStarted","Data":"3f0daca76136e111bcd093c8392ee1baef207a5d78065d45a2b465f4f1a48332"}
Nov 02 14:30:13.508123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:13.508075    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qkbsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 14:30:13.508343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:13.508134    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 14:30:14.112521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:14.112473    5034 generic.go:356] "Generic (PLEG): container finished" podID="f451c332-f287-45c4-8286-9844268d9bd6" containerID="24ad91f19c37de8838ffdfa05d97449bce54df3376b7547a23330985a13d3c1f" exitCode=0
Nov 02 14:30:14.112932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:14.112617    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn" event={"ID":"f451c332-f287-45c4-8286-9844268d9bd6","Type":"ContainerDied","Data":"24ad91f19c37de8838ffdfa05d97449bce54df3376b7547a23330985a13d3c1f"}
Nov 02 14:30:14.508200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:14.508149    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qkbsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 14:30:14.508367 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:14.508211    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 14:30:15.356338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.356303    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:15.449161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.449060    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-xks-ca\") pod \"f451c332-f287-45c4-8286-9844268d9bd6\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") "
Nov 02 14:30:15.449556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.449438    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-openshift-ca\") pod \"f451c332-f287-45c4-8286-9844268d9bd6\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") "
Nov 02 14:30:15.449556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.449482    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f451c332-f287-45c4-8286-9844268d9bd6-kube-api-access-c8zgn\" (UniqueName: \"kubernetes.io/projected/f451c332-f287-45c4-8286-9844268d9bd6-kube-api-access-c8zgn\") pod \"f451c332-f287-45c4-8286-9844268d9bd6\" (UID: \"f451c332-f287-45c4-8286-9844268d9bd6\") "
Nov 02 14:30:15.450381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.450353    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-openshift-ca" pod "f451c332-f287-45c4-8286-9844268d9bd6" (UID: "f451c332-f287-45c4-8286-9844268d9bd6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:15.452927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.452878    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-xks-ca" pod "f451c332-f287-45c4-8286-9844268d9bd6" (UID: "f451c332-f287-45c4-8286-9844268d9bd6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:30:15.453062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.453017    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f451c332-f287-45c4-8286-9844268d9bd6-kube-api-access-c8zgn" pod "f451c332-f287-45c4-8286-9844268d9bd6" (UID: "f451c332-f287-45c4-8286-9844268d9bd6"). InnerVolumeSpecName "kube-api-access-c8zgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:15.507755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.507718    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qkbsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused" start-of-body=
Nov 02 14:30:15.508033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.507788    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": dial tcp 10.132.1.66:15021: connect: connection refused"
Nov 02 14:30:15.552430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.552401    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:15.552430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.552432    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f451c332-f287-45c4-8286-9844268d9bd6-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:15.552430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:15.552446    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8zgn\" (UniqueName: \"kubernetes.io/projected/f451c332-f287-45c4-8286-9844268d9bd6-kube-api-access-c8zgn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:16.137304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:16.137277    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"
Nov 02 14:30:16.137530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:16.137300    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn" event={"ID":"f451c332-f287-45c4-8286-9844268d9bd6","Type":"ContainerDied","Data":"3f0daca76136e111bcd093c8392ee1baef207a5d78065d45a2b465f4f1a48332"}
Nov 02 14:30:16.137530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:16.137361    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f0daca76136e111bcd093c8392ee1baef207a5d78065d45a2b465f4f1a48332"
Nov 02 14:30:16.387659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:16.387555    5034 scope.go:122] "RemoveContainer" containerID="e041a6f5e9e5bbfed029fd9b7cddda9daeace8936d7b30169612a12c9a4458b7"
Nov 02 14:30:16.388247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:16.387943    5034 scope.go:122] "RemoveContainer" containerID="e981499680e7e43459fbc9b608cca5a2edb2c632de258456c3c4c8ba86e451c8"
Nov 02 14:30:17.152622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.152589    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" event={"ID":"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68","Type":"ContainerStarted","Data":"8ba114451b95d72edabff4dc481b6f2e0cef4b245b030c550adbb35ce40d870b"}
Nov 02 14:30:17.155204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.155171    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" event={"ID":"00f54dd3-b21f-4705-af6f-d29c35b9368c","Type":"ContainerStarted","Data":"0338a8ffc4fa3dc7385452ba6266bb3e8aa6852736a707e8e55c7282462dcd29"}
Nov 02 14:30:17.508623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.508580    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-qkbsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:30:17.509117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.508647    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:30:17.567406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.567375    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:17.568125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.568101    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:17.577204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.577140    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:30:17.805448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.805359    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"]
Nov 02 14:30:17.806881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.806857    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f451c332-f287-45c4-8286-9844268d9bd6" containerName="revoke-keys"
Nov 02 14:30:17.806962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.806884    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f451c332-f287-45c4-8286-9844268d9bd6" containerName="revoke-keys"
Nov 02 14:30:17.807243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.807229    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f451c332-f287-45c4-8286-9844268d9bd6" containerName="revoke-keys"
Nov 02 14:30:17.812418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.812394    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:17.848604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.848572    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:30:17.849038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.849014    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"]
Nov 02 14:30:17.912105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.912051    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b91daf-shmbb\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:17.912473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.912444    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b91daf-shmbb\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:17.912541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:17.912496    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5g29\" (UniqueName: \"kubernetes.io/projected/d14778a4-3528-4a2c-b61c-f7ae977edeab-kube-api-access-p5g29\") pod \"maas-api-revoke-sub-70b91daf-shmbb\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:18.014508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.014463    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b91daf-shmbb\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:18.014751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.014715    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b91daf-shmbb\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:18.014906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.014767    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5g29\" (UniqueName: \"kubernetes.io/projected/d14778a4-3528-4a2c-b61c-f7ae977edeab-kube-api-access-p5g29\") pod \"maas-api-revoke-sub-70b91daf-shmbb\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:18.015605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.015578    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b91daf-shmbb\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:18.017047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.017028    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b91daf-shmbb\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:18.040669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.040636    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5g29\" (UniqueName: \"kubernetes.io/projected/d14778a4-3528-4a2c-b61c-f7ae977edeab-kube-api-access-p5g29\") pod \"maas-api-revoke-sub-70b91daf-shmbb\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:18.130867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.130452    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:18.176960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.176913    5034 generic.go:356] "Generic (PLEG): container finished" podID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerID="0338a8ffc4fa3dc7385452ba6266bb3e8aa6852736a707e8e55c7282462dcd29" exitCode=60
Nov 02 14:30:18.177098 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.176985    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" event={"ID":"00f54dd3-b21f-4705-af6f-d29c35b9368c","Type":"ContainerDied","Data":"0338a8ffc4fa3dc7385452ba6266bb3e8aa6852736a707e8e55c7282462dcd29"}
Nov 02 14:30:18.177098 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.177059    5034 scope.go:122] "RemoveContainer" containerID="e041a6f5e9e5bbfed029fd9b7cddda9daeace8936d7b30169612a12c9a4458b7"
Nov 02 14:30:18.177610 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.177582    5034 scope.go:122] "RemoveContainer" containerID="0338a8ffc4fa3dc7385452ba6266bb3e8aa6852736a707e8e55c7282462dcd29"
Nov 02 14:30:18.178092 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:18.178068    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48_odh-ai-gateway-infra(00f54dd3-b21f-4705-af6f-d29c35b9368c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c"
Nov 02 14:30:18.182467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.182187    5034 generic.go:356] "Generic (PLEG): container finished" podID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerID="8ba114451b95d72edabff4dc481b6f2e0cef4b245b030c550adbb35ce40d870b" exitCode=60
Nov 02 14:30:18.182467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.182268    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" event={"ID":"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68","Type":"ContainerDied","Data":"8ba114451b95d72edabff4dc481b6f2e0cef4b245b030c550adbb35ce40d870b"}
Nov 02 14:30:18.182944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.182823    5034 scope.go:122] "RemoveContainer" containerID="8ba114451b95d72edabff4dc481b6f2e0cef4b245b030c550adbb35ce40d870b"
Nov 02 14:30:18.183571 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:18.183230    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh_odh-ai-gateway-infra(d1bf66ee-0739-4042-9f1d-d1f0f5de7f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68"
Nov 02 14:30:18.204993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.204967    5034 scope.go:122] "RemoveContainer" containerID="e981499680e7e43459fbc9b608cca5a2edb2c632de258456c3c4c8ba86e451c8"
Nov 02 14:30:18.240435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.240404    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"]
Nov 02 14:30:18.295242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.295064    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"]
Nov 02 14:30:18.387232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.387136    5034 scope.go:122] "RemoveContainer" containerID="ac021091e086b80016555e177ce22f9759825fa8b714d428f52c336c5cbde16d"
Nov 02 14:30:18.387390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.387361    5034 scope.go:122] "RemoveContainer" containerID="3839801e3c67917ee747645cf88d334ce9f8448c259b037f363e794e3f9740bf"
Nov 02 14:30:18.387563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.387541    5034 scope.go:122] "RemoveContainer" containerID="22c1b8c3c306dd3bd0e27f76cf755fcaaa71285cf48eed7e8bf8ea9ff2b97e6c"
Nov 02 14:30:18.388727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.387978    5034 scope.go:122] "RemoveContainer" containerID="1db6c8844fcecca73dea977079c95c2dffa2e090f23d39fc50265fbe3227a530"
Nov 02 14:30:18.427801 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:18.427705    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:30:18.489919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:18.489882    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"]
Nov 02 14:30:18.589318 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:18.589275    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14778a4_3528_4a2c_b61c_f7ae977edeab.slice/crio-cb76ab978a785bdb89aadc6c1677bd801940c04518fb903e25fb726f31f315a9 WatchSource:0}: Error finding container cb76ab978a785bdb89aadc6c1677bd801940c04518fb903e25fb726f31f315a9: Status 404 returned error can't find the container with id cb76ab978a785bdb89aadc6c1677bd801940c04518fb903e25fb726f31f315a9
Nov 02 14:30:19.198396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.198355    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" event={"ID":"798e455b-975d-468b-bc10-3e020e992626","Type":"ContainerStarted","Data":"e1b89eff865b5630fdc5b24e140e70ba9d0205aba773d30eba6682d7fcebab09"}
Nov 02 14:30:19.202550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.202473    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb" event={"ID":"d14778a4-3528-4a2c-b61c-f7ae977edeab","Type":"ContainerStarted","Data":"b5649419fdebdd510bec8b13ddcd1deead5dbb0a521b7cfded397fe9393c1996"}
Nov 02 14:30:19.202550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.202525    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb" event={"ID":"d14778a4-3528-4a2c-b61c-f7ae977edeab","Type":"ContainerStarted","Data":"cb76ab978a785bdb89aadc6c1677bd801940c04518fb903e25fb726f31f315a9"}
Nov 02 14:30:19.205882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.205852    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" event={"ID":"110008e0-3d52-4923-9188-98bafbfccf54","Type":"ContainerStarted","Data":"4bfcff7ed20416575ba878f60fe6c3e83cacc80d40bd62737cf7b662db3e3393"}
Nov 02 14:30:19.208266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.208236    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" event={"ID":"1c667d91-1fd4-438c-b3a3-8281cf003b88","Type":"ContainerStarted","Data":"59ee1b7053593485dfff050075b33777e7e7463aef1b563fc81836df847349f9"}
Nov 02 14:30:19.216605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.216570    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" event={"ID":"ce138676-ad2e-4761-ad72-571e0b3bf23c","Type":"ContainerStarted","Data":"62cb8fec03b8d2d660b5c7537ad044e37eb4fa182d42b5d8e10a1d4a7217bbd4"}
Nov 02 14:30:19.306812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.306211    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" podStartSLOduration=19.306191156 podStartE2EDuration="19.306191156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:19.288040262 +0000 UTC m=+2810.039344701" watchObservedRunningTime="2026-11-02 14:30:19.306191156 +0000 UTC m=+2810.057495595"
Nov 02 14:30:19.389641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.389598    5034 scope.go:122] "RemoveContainer" containerID="bc56e824ae5d33698461bbe334d9fdbdd3d3d3ac4ad4b0c8b1302c2061751726"
Nov 02 14:30:19.394795 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.394746    5034 scope.go:122] "RemoveContainer" containerID="b99e0de451512811d3cb8b8cfd7e54fe15ad6901af49dd0f9468dde948c089db"
Nov 02 14:30:19.404798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.404745    5034 scope.go:122] "RemoveContainer" containerID="c7ae8aac0380e188698b7be3d8888966f53c538ee3f1e469aa92b12ab9729a14"
Nov 02 14:30:19.408113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.407363    5034 scope.go:122] "RemoveContainer" containerID="a4b5bedbc930eac645478b2f9463077508bce65ee67b18240933f519baedb209"
Nov 02 14:30:19.408113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.408058    5034 scope.go:122] "RemoveContainer" containerID="da88c65717ea701feb56a6cc7f7a0862165ca790a166172f3d0a0408787a7d00"
Nov 02 14:30:19.478999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.478410    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:19.551231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.550990    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb" podStartSLOduration=2.5509752089999997 podStartE2EDuration="2.550975209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:19.549192769 +0000 UTC m=+2810.300497207" watchObservedRunningTime="2026-11-02 14:30:19.550975209 +0000 UTC m=+2810.302279682"
Nov 02 14:30:19.612002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.609680    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f54dd3-b21f-4705-af6f-d29c35b9368c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f54dd3-b21f-4705-af6f-d29c35b9368c-maas-api-service-ca\") pod \"00f54dd3-b21f-4705-af6f-d29c35b9368c\" (UID: \"00f54dd3-b21f-4705-af6f-d29c35b9368c\") "
Nov 02 14:30:19.629175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.629140    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f54dd3-b21f-4705-af6f-d29c35b9368c-kube-api-access-vtq7s\" (UniqueName: \"kubernetes.io/projected/00f54dd3-b21f-4705-af6f-d29c35b9368c-kube-api-access-vtq7s\") pod \"00f54dd3-b21f-4705-af6f-d29c35b9368c\" (UID: \"00f54dd3-b21f-4705-af6f-d29c35b9368c\") "
Nov 02 14:30:19.633436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.633391    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f54dd3-b21f-4705-af6f-d29c35b9368c-maas-api-service-ca" pod "00f54dd3-b21f-4705-af6f-d29c35b9368c" (UID: "00f54dd3-b21f-4705-af6f-d29c35b9368c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:19.638715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.638679    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f54dd3-b21f-4705-af6f-d29c35b9368c-kube-api-access-vtq7s" pod "00f54dd3-b21f-4705-af6f-d29c35b9368c" (UID: "00f54dd3-b21f-4705-af6f-d29c35b9368c"). InnerVolumeSpecName "kube-api-access-vtq7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:19.659030 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:19.658990    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce138676_ad2e_4761_ad72_571e0b3bf23c.slice/crio-conmon-62cb8fec03b8d2d660b5c7537ad044e37eb4fa182d42b5d8e10a1d4a7217bbd4.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 14:30:19.682828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.682797    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:19.735725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.731583    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-maas-api-service-ca\") pod \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\" (UID: \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\") "
Nov 02 14:30:19.735725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.731881    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-kube-api-access-wt8r5\" (UniqueName: \"kubernetes.io/projected/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-kube-api-access-wt8r5\") pod \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\" (UID: \"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68\") "
Nov 02 14:30:19.735725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.732695    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00f54dd3-b21f-4705-af6f-d29c35b9368c-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:19.735725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.732721    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtq7s\" (UniqueName: \"kubernetes.io/projected/00f54dd3-b21f-4705-af6f-d29c35b9368c-kube-api-access-vtq7s\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:19.735725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.734159    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-maas-api-service-ca" pod "d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" (UID: "d1bf66ee-0739-4042-9f1d-d1f0f5de7f68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:19.775741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.775701    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-kube-api-access-wt8r5" pod "d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" (UID: "d1bf66ee-0739-4042-9f1d-d1f0f5de7f68"). InnerVolumeSpecName "kube-api-access-wt8r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:19.835643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.835516    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt8r5\" (UniqueName: \"kubernetes.io/projected/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-kube-api-access-wt8r5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:19.835643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:19.835563    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:20.228434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.228402    5034 generic.go:356] "Generic (PLEG): container finished" podID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerID="59ee1b7053593485dfff050075b33777e7e7463aef1b563fc81836df847349f9" exitCode=60
Nov 02 14:30:20.228626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.228479    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" event={"ID":"1c667d91-1fd4-438c-b3a3-8281cf003b88","Type":"ContainerDied","Data":"59ee1b7053593485dfff050075b33777e7e7463aef1b563fc81836df847349f9"}
Nov 02 14:30:20.228626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.228550    5034 scope.go:122] "RemoveContainer" containerID="22c1b8c3c306dd3bd0e27f76cf755fcaaa71285cf48eed7e8bf8ea9ff2b97e6c"
Nov 02 14:30:20.229142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.229095    5034 scope.go:122] "RemoveContainer" containerID="59ee1b7053593485dfff050075b33777e7e7463aef1b563fc81836df847349f9"
Nov 02 14:30:20.229876 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:20.229525    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg_odh-ai-gateway-infra(1c667d91-1fd4-438c-b3a3-8281cf003b88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88"
Nov 02 14:30:20.230218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.230203    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"
Nov 02 14:30:20.230293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.230239    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh" event={"ID":"d1bf66ee-0739-4042-9f1d-d1f0f5de7f68","Type":"ContainerDied","Data":"b861818721e0f0233512546415b90bf26a7d2d5a9fe027366357cb8cbb7fd16a"}
Nov 02 14:30:20.232321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.232294    5034 generic.go:356] "Generic (PLEG): container finished" podID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerID="62cb8fec03b8d2d660b5c7537ad044e37eb4fa182d42b5d8e10a1d4a7217bbd4" exitCode=60
Nov 02 14:30:20.232403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.232358    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" event={"ID":"ce138676-ad2e-4761-ad72-571e0b3bf23c","Type":"ContainerDied","Data":"62cb8fec03b8d2d660b5c7537ad044e37eb4fa182d42b5d8e10a1d4a7217bbd4"}
Nov 02 14:30:20.232763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.232744    5034 scope.go:122] "RemoveContainer" containerID="62cb8fec03b8d2d660b5c7537ad044e37eb4fa182d42b5d8e10a1d4a7217bbd4"
Nov 02 14:30:20.233096 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:20.233076    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw_odh-ai-gateway-infra(ce138676-ad2e-4761-ad72-571e0b3bf23c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c"
Nov 02 14:30:20.234610 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.234585    5034 generic.go:356] "Generic (PLEG): container finished" podID="798e455b-975d-468b-bc10-3e020e992626" containerID="e1b89eff865b5630fdc5b24e140e70ba9d0205aba773d30eba6682d7fcebab09" exitCode=60
Nov 02 14:30:20.234721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.234691    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" event={"ID":"798e455b-975d-468b-bc10-3e020e992626","Type":"ContainerDied","Data":"e1b89eff865b5630fdc5b24e140e70ba9d0205aba773d30eba6682d7fcebab09"}
Nov 02 14:30:20.235457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.235435    5034 scope.go:122] "RemoveContainer" containerID="e1b89eff865b5630fdc5b24e140e70ba9d0205aba773d30eba6682d7fcebab09"
Nov 02 14:30:20.235829 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:20.235806    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl_odh-ai-gateway-infra(798e455b-975d-468b-bc10-3e020e992626)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" podUID="798e455b-975d-468b-bc10-3e020e992626"
Nov 02 14:30:20.236423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.236407    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"
Nov 02 14:30:20.236524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.236423    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48" event={"ID":"00f54dd3-b21f-4705-af6f-d29c35b9368c","Type":"ContainerDied","Data":"159572d135b1d6f3cabfaf033ff8dc95ad5d3d6b5645c15e9eacef281184e624"}
Nov 02 14:30:20.238307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.238281    5034 generic.go:356] "Generic (PLEG): container finished" podID="d14778a4-3528-4a2c-b61c-f7ae977edeab" containerID="b5649419fdebdd510bec8b13ddcd1deead5dbb0a521b7cfded397fe9393c1996" exitCode=0
Nov 02 14:30:20.238415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.238398    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb" event={"ID":"d14778a4-3528-4a2c-b61c-f7ae977edeab","Type":"ContainerDied","Data":"b5649419fdebdd510bec8b13ddcd1deead5dbb0a521b7cfded397fe9393c1996"}
Nov 02 14:30:20.242089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.242064    5034 generic.go:356] "Generic (PLEG): container finished" podID="110008e0-3d52-4923-9188-98bafbfccf54" containerID="4bfcff7ed20416575ba878f60fe6c3e83cacc80d40bd62737cf7b662db3e3393" exitCode=60
Nov 02 14:30:20.242261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.242245    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" event={"ID":"110008e0-3d52-4923-9188-98bafbfccf54","Type":"ContainerDied","Data":"4bfcff7ed20416575ba878f60fe6c3e83cacc80d40bd62737cf7b662db3e3393"}
Nov 02 14:30:20.242624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.242603    5034 scope.go:122] "RemoveContainer" containerID="4bfcff7ed20416575ba878f60fe6c3e83cacc80d40bd62737cf7b662db3e3393"
Nov 02 14:30:20.242947 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:20.242921    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj_odh-ai-gateway-infra(110008e0-3d52-4923-9188-98bafbfccf54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" podUID="110008e0-3d52-4923-9188-98bafbfccf54"
Nov 02 14:30:20.247008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.246988    5034 scope.go:122] "RemoveContainer" containerID="8ba114451b95d72edabff4dc481b6f2e0cef4b245b030c550adbb35ce40d870b"
Nov 02 14:30:20.258526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.258505    5034 scope.go:122] "RemoveContainer" containerID="1db6c8844fcecca73dea977079c95c2dffa2e090f23d39fc50265fbe3227a530"
Nov 02 14:30:20.270623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.270593    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"]
Nov 02 14:30:20.288394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.288323    5034 scope.go:122] "RemoveContainer" containerID="ac021091e086b80016555e177ce22f9759825fa8b714d428f52c336c5cbde16d"
Nov 02 14:30:20.311910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.311883    5034 scope.go:122] "RemoveContainer" containerID="0338a8ffc4fa3dc7385452ba6266bb3e8aa6852736a707e8e55c7282462dcd29"
Nov 02 14:30:20.312689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.312663    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"]
Nov 02 14:30:20.324610 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.324585    5034 scope.go:122] "RemoveContainer" containerID="3839801e3c67917ee747645cf88d334ce9f8448c259b037f363e794e3f9740bf"
Nov 02 14:30:20.354529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.354500    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"]
Nov 02 14:30:20.387178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.387147    5034 scope.go:122] "RemoveContainer" containerID="026b3728a2cf6f7d752de912d9f60c5f606da9bbacd0ff104738a11cc4624db9"
Nov 02 14:30:20.387906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.387360    5034 scope.go:122] "RemoveContainer" containerID="c1538081a0a64c0264147e90bee1be2d56972d6ccba4608276f733a20114db68"
Nov 02 14:30:20.388178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.388160    5034 scope.go:122] "RemoveContainer" containerID="759aff80ff571981c038a04af584cd3d57b2aad37839c1c4e433fce468b94688"
Nov 02 14:30:20.423591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.423540    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"]
Nov 02 14:30:20.484087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.484056    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"]
Nov 02 14:30:20.499120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.499085    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-87efd3-29893830-f6shh"]
Nov 02 14:30:20.538007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.537967    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"]
Nov 02 14:30:20.546486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:20.546446    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ee3eca-29893830-hfh48"]
Nov 02 14:30:21.256860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.256821    5034 generic.go:356] "Generic (PLEG): container finished" podID="5391b42a-6255-4531-957f-31dd8616f975" containerID="0204d48376e0ef25e79d88d8d8fbec557c85d9316b15b5dbdd4c0b7493ff7025" exitCode=60
Nov 02 14:30:21.257262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.256890    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" event={"ID":"5391b42a-6255-4531-957f-31dd8616f975","Type":"ContainerDied","Data":"0204d48376e0ef25e79d88d8d8fbec557c85d9316b15b5dbdd4c0b7493ff7025"}
Nov 02 14:30:21.257262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.256946    5034 scope.go:122] "RemoveContainer" containerID="b99e0de451512811d3cb8b8cfd7e54fe15ad6901af49dd0f9468dde948c089db"
Nov 02 14:30:21.257522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.257499    5034 scope.go:122] "RemoveContainer" containerID="0204d48376e0ef25e79d88d8d8fbec557c85d9316b15b5dbdd4c0b7493ff7025"
Nov 02 14:30:21.258058 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:21.258002    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2_odh-ai-gateway-infra(5391b42a-6255-4531-957f-31dd8616f975)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" podUID="5391b42a-6255-4531-957f-31dd8616f975"
Nov 02 14:30:21.261091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.261067    5034 generic.go:356] "Generic (PLEG): container finished" podID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerID="e0d059b546bf734ef1632ca024d7f837a876d306497e9ddd231d584443663b94" exitCode=60
Nov 02 14:30:21.261171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.261140    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" event={"ID":"6d7b00a2-30e2-4c1f-bc50-d98158707200","Type":"ContainerDied","Data":"e0d059b546bf734ef1632ca024d7f837a876d306497e9ddd231d584443663b94"}
Nov 02 14:30:21.261661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.261642    5034 scope.go:122] "RemoveContainer" containerID="e0d059b546bf734ef1632ca024d7f837a876d306497e9ddd231d584443663b94"
Nov 02 14:30:21.262141 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:21.262114    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7_odh-ai-gateway-infra(6d7b00a2-30e2-4c1f-bc50-d98158707200)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200"
Nov 02 14:30:21.267001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.266095    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" event={"ID":"9a96c9cc-4506-402a-8923-620b9381ecc9","Type":"ContainerStarted","Data":"4fb5a15dedbd3cad6d6c9365c6278eef142a4d7fe71efc7f30c6213b320e009b"}
Nov 02 14:30:21.275812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.275468    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" event={"ID":"786d86b7-0109-4fe1-abce-3960750cc80d","Type":"ContainerStarted","Data":"722e5903a60e4a60799100167456d0f981329414663202fa532c15eacaf701e5"}
Nov 02 14:30:21.280078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.279488    5034 generic.go:356] "Generic (PLEG): container finished" podID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerID="861b49bfffa13cdbf4f95990ba6bac26962029d821926309617b2f3a12b3a6c2" exitCode=60
Nov 02 14:30:21.280078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.279595    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" event={"ID":"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd","Type":"ContainerDied","Data":"861b49bfffa13cdbf4f95990ba6bac26962029d821926309617b2f3a12b3a6c2"}
Nov 02 14:30:21.280242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.280156    5034 scope.go:122] "RemoveContainer" containerID="861b49bfffa13cdbf4f95990ba6bac26962029d821926309617b2f3a12b3a6c2"
Nov 02 14:30:21.280658 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:21.280633    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4_odh-ai-gateway-infra(ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd"
Nov 02 14:30:21.282486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.282462    5034 scope.go:122] "RemoveContainer" containerID="da88c65717ea701feb56a6cc7f7a0862165ca790a166172f3d0a0408787a7d00"
Nov 02 14:30:21.284499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.284469    5034 generic.go:356] "Generic (PLEG): container finished" podID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerID="ef8cff6d45454de5012bf5a031232a3f881d7db2239b14df89afbef68827d497" exitCode=60
Nov 02 14:30:21.284665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.284600    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" event={"ID":"1c6427b3-906d-47bc-938f-c38f0c1b5b5a","Type":"ContainerDied","Data":"ef8cff6d45454de5012bf5a031232a3f881d7db2239b14df89afbef68827d497"}
Nov 02 14:30:21.285942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.285919    5034 scope.go:122] "RemoveContainer" containerID="ef8cff6d45454de5012bf5a031232a3f881d7db2239b14df89afbef68827d497"
Nov 02 14:30:21.286429 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:21.286397    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd_odh-ai-gateway-infra(1c6427b3-906d-47bc-938f-c38f0c1b5b5a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a"
Nov 02 14:30:21.290452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.290347    5034 generic.go:356] "Generic (PLEG): container finished" podID="12c10b34-f265-434c-a511-077efe1688d0" containerID="f4db173ae6b77c6276066ba8d5c11208f78456769bf3258fe75c606bc39455d8" exitCode=60
Nov 02 14:30:21.290543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.290467    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" event={"ID":"12c10b34-f265-434c-a511-077efe1688d0","Type":"ContainerDied","Data":"f4db173ae6b77c6276066ba8d5c11208f78456769bf3258fe75c606bc39455d8"}
Nov 02 14:30:21.291037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.290999    5034 scope.go:122] "RemoveContainer" containerID="f4db173ae6b77c6276066ba8d5c11208f78456769bf3258fe75c606bc39455d8"
Nov 02 14:30:21.302004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.299583    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" event={"ID":"732c703d-dd22-4a0b-a2c0-93e2ac208979","Type":"ContainerStarted","Data":"0f2927942e35af93cbe2144b1ce5af5bf38b5f0dc7d6e77b86c521964ee5ec68"}
Nov 02 14:30:21.307684 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:21.307652    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb_odh-ai-gateway-infra(12c10b34-f265-434c-a511-077efe1688d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" podUID="12c10b34-f265-434c-a511-077efe1688d0"
Nov 02 14:30:21.358014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.357964    5034 scope.go:122] "RemoveContainer" containerID="a4b5bedbc930eac645478b2f9463077508bce65ee67b18240933f519baedb209"
Nov 02 14:30:21.431162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.430067    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" path="/var/lib/kubelet/pods/00f54dd3-b21f-4705-af6f-d29c35b9368c/volumes"
Nov 02 14:30:21.431162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.430904    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" path="/var/lib/kubelet/pods/d1bf66ee-0739-4042-9f1d-d1f0f5de7f68/volumes"
Nov 02 14:30:21.433706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.433679    5034 scope.go:122] "RemoveContainer" containerID="bc56e824ae5d33698461bbe334d9fdbdd3d3d3ac4ad4b0c8b1302c2061751726"
Nov 02 14:30:21.467707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.467617    5034 scope.go:122] "RemoveContainer" containerID="c7ae8aac0380e188698b7be3d8888966f53c538ee3f1e469aa92b12ab9729a14"
Nov 02 14:30:21.596560 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.596527    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:21.693188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.691338    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c667d91-1fd4-438c-b3a3-8281cf003b88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c667d91-1fd4-438c-b3a3-8281cf003b88-maas-api-service-ca\") pod \"1c667d91-1fd4-438c-b3a3-8281cf003b88\" (UID: \"1c667d91-1fd4-438c-b3a3-8281cf003b88\") "
Nov 02 14:30:21.693188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.691400    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c667d91-1fd4-438c-b3a3-8281cf003b88-kube-api-access-g8wx6\" (UniqueName: \"kubernetes.io/projected/1c667d91-1fd4-438c-b3a3-8281cf003b88-kube-api-access-g8wx6\") pod \"1c667d91-1fd4-438c-b3a3-8281cf003b88\" (UID: \"1c667d91-1fd4-438c-b3a3-8281cf003b88\") "
Nov 02 14:30:21.693650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.693598    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c667d91-1fd4-438c-b3a3-8281cf003b88-maas-api-service-ca" pod "1c667d91-1fd4-438c-b3a3-8281cf003b88" (UID: "1c667d91-1fd4-438c-b3a3-8281cf003b88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:21.730300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.730252    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c667d91-1fd4-438c-b3a3-8281cf003b88-kube-api-access-g8wx6" pod "1c667d91-1fd4-438c-b3a3-8281cf003b88" (UID: "1c667d91-1fd4-438c-b3a3-8281cf003b88"). InnerVolumeSpecName "kube-api-access-g8wx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:21.797415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.795703    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c667d91-1fd4-438c-b3a3-8281cf003b88-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:21.797415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.795760    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8wx6\" (UniqueName: \"kubernetes.io/projected/1c667d91-1fd4-438c-b3a3-8281cf003b88-kube-api-access-g8wx6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:21.908112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.908016    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:21.920985 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.920954    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:21.924520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.924494    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:21.933916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.933889    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:21.998624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.998585    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798e455b-975d-468b-bc10-3e020e992626-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798e455b-975d-468b-bc10-3e020e992626-maas-api-service-ca\") pod \"798e455b-975d-468b-bc10-3e020e992626\" (UID: \"798e455b-975d-468b-bc10-3e020e992626\") "
Nov 02 14:30:21.998965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.998945    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-openshift-ca\") pod \"d14778a4-3528-4a2c-b61c-f7ae977edeab\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") "
Nov 02 14:30:21.999149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.999121    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110008e0-3d52-4923-9188-98bafbfccf54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110008e0-3d52-4923-9188-98bafbfccf54-maas-api-service-ca\") pod \"110008e0-3d52-4923-9188-98bafbfccf54\" (UID: \"110008e0-3d52-4923-9188-98bafbfccf54\") "
Nov 02 14:30:21.999283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.999263    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-xks-ca\") pod \"d14778a4-3528-4a2c-b61c-f7ae977edeab\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") "
Nov 02 14:30:21.999346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.999336    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce138676-ad2e-4761-ad72-571e0b3bf23c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce138676-ad2e-4761-ad72-571e0b3bf23c-maas-api-service-ca\") pod \"ce138676-ad2e-4761-ad72-571e0b3bf23c\" (UID: \"ce138676-ad2e-4761-ad72-571e0b3bf23c\") "
Nov 02 14:30:21.999435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.999415    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce138676-ad2e-4761-ad72-571e0b3bf23c-kube-api-access-hnbvw\" (UniqueName: \"kubernetes.io/projected/ce138676-ad2e-4761-ad72-571e0b3bf23c-kube-api-access-hnbvw\") pod \"ce138676-ad2e-4761-ad72-571e0b3bf23c\" (UID: \"ce138676-ad2e-4761-ad72-571e0b3bf23c\") "
Nov 02 14:30:21.999691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.999670    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798e455b-975d-468b-bc10-3e020e992626-kube-api-access-bgcbm\" (UniqueName: \"kubernetes.io/projected/798e455b-975d-468b-bc10-3e020e992626-kube-api-access-bgcbm\") pod \"798e455b-975d-468b-bc10-3e020e992626\" (UID: \"798e455b-975d-468b-bc10-3e020e992626\") "
Nov 02 14:30:21.999877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.999733    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d14778a4-3528-4a2c-b61c-f7ae977edeab-kube-api-access-p5g29\" (UniqueName: \"kubernetes.io/projected/d14778a4-3528-4a2c-b61c-f7ae977edeab-kube-api-access-p5g29\") pod \"d14778a4-3528-4a2c-b61c-f7ae977edeab\" (UID: \"d14778a4-3528-4a2c-b61c-f7ae977edeab\") "
Nov 02 14:30:21.999984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:21.999933    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110008e0-3d52-4923-9188-98bafbfccf54-kube-api-access-z4dkq\" (UniqueName: \"kubernetes.io/projected/110008e0-3d52-4923-9188-98bafbfccf54-kube-api-access-z4dkq\") pod \"110008e0-3d52-4923-9188-98bafbfccf54\" (UID: \"110008e0-3d52-4923-9188-98bafbfccf54\") "
Nov 02 14:30:22.002629 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.002329    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110008e0-3d52-4923-9188-98bafbfccf54-maas-api-service-ca" pod "110008e0-3d52-4923-9188-98bafbfccf54" (UID: "110008e0-3d52-4923-9188-98bafbfccf54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:22.003219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.002920    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce138676-ad2e-4761-ad72-571e0b3bf23c-maas-api-service-ca" pod "ce138676-ad2e-4761-ad72-571e0b3bf23c" (UID: "ce138676-ad2e-4761-ad72-571e0b3bf23c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:22.003808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.003437    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798e455b-975d-468b-bc10-3e020e992626-maas-api-service-ca" pod "798e455b-975d-468b-bc10-3e020e992626" (UID: "798e455b-975d-468b-bc10-3e020e992626"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:22.004292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.004262    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-openshift-ca" pod "d14778a4-3528-4a2c-b61c-f7ae977edeab" (UID: "d14778a4-3528-4a2c-b61c-f7ae977edeab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:22.033838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.033397    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-xks-ca" pod "d14778a4-3528-4a2c-b61c-f7ae977edeab" (UID: "d14778a4-3528-4a2c-b61c-f7ae977edeab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:30:22.033838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.033451    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce138676-ad2e-4761-ad72-571e0b3bf23c-kube-api-access-hnbvw" pod "ce138676-ad2e-4761-ad72-571e0b3bf23c" (UID: "ce138676-ad2e-4761-ad72-571e0b3bf23c"). InnerVolumeSpecName "kube-api-access-hnbvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:22.033838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.033545    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110008e0-3d52-4923-9188-98bafbfccf54-kube-api-access-z4dkq" pod "110008e0-3d52-4923-9188-98bafbfccf54" (UID: "110008e0-3d52-4923-9188-98bafbfccf54"). InnerVolumeSpecName "kube-api-access-z4dkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:22.036828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.034951    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d14778a4-3528-4a2c-b61c-f7ae977edeab-kube-api-access-p5g29" pod "d14778a4-3528-4a2c-b61c-f7ae977edeab" (UID: "d14778a4-3528-4a2c-b61c-f7ae977edeab"). InnerVolumeSpecName "kube-api-access-p5g29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:22.047104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.045040    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798e455b-975d-468b-bc10-3e020e992626-kube-api-access-bgcbm" pod "798e455b-975d-468b-bc10-3e020e992626" (UID: "798e455b-975d-468b-bc10-3e020e992626"). InnerVolumeSpecName "kube-api-access-bgcbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:22.107489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.107399    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce138676-ad2e-4761-ad72-571e0b3bf23c-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:22.107489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.107440    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnbvw\" (UniqueName: \"kubernetes.io/projected/ce138676-ad2e-4761-ad72-571e0b3bf23c-kube-api-access-hnbvw\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:22.107489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.107455    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgcbm\" (UniqueName: \"kubernetes.io/projected/798e455b-975d-468b-bc10-3e020e992626-kube-api-access-bgcbm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:22.107489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.107469    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5g29\" (UniqueName: \"kubernetes.io/projected/d14778a4-3528-4a2c-b61c-f7ae977edeab-kube-api-access-p5g29\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:22.107489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.107483    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4dkq\" (UniqueName: \"kubernetes.io/projected/110008e0-3d52-4923-9188-98bafbfccf54-kube-api-access-z4dkq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:22.107824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.107497    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/798e455b-975d-468b-bc10-3e020e992626-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:22.107824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.107510    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:22.107824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.107527    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110008e0-3d52-4923-9188-98bafbfccf54-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:22.107824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.107542    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d14778a4-3528-4a2c-b61c-f7ae977edeab-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:22.316973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.316938    5034 generic.go:356] "Generic (PLEG): container finished" podID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerID="0f2927942e35af93cbe2144b1ce5af5bf38b5f0dc7d6e77b86c521964ee5ec68" exitCode=60
Nov 02 14:30:22.317441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.317019    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" event={"ID":"732c703d-dd22-4a0b-a2c0-93e2ac208979","Type":"ContainerDied","Data":"0f2927942e35af93cbe2144b1ce5af5bf38b5f0dc7d6e77b86c521964ee5ec68"}
Nov 02 14:30:22.317441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.317128    5034 scope.go:122] "RemoveContainer" containerID="c1538081a0a64c0264147e90bee1be2d56972d6ccba4608276f733a20114db68"
Nov 02 14:30:22.317564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.317546    5034 scope.go:122] "RemoveContainer" containerID="0f2927942e35af93cbe2144b1ce5af5bf38b5f0dc7d6e77b86c521964ee5ec68"
Nov 02 14:30:22.317948 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:22.317923    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9_odh-ai-gateway-infra(732c703d-dd22-4a0b-a2c0-93e2ac208979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979"
Nov 02 14:30:22.319294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.319267    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"
Nov 02 14:30:22.319390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.319300    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg" event={"ID":"1c667d91-1fd4-438c-b3a3-8281cf003b88","Type":"ContainerDied","Data":"1b61f60e0cd00cf8f3d70583160d86da6c92ea32bc33861678ae682787bde3af"}
Nov 02 14:30:22.322417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.322396    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw" event={"ID":"ce138676-ad2e-4761-ad72-571e0b3bf23c","Type":"ContainerDied","Data":"fc25122adf47f8d5a3a785c6c84433af27d21040f67c33cfd5ac72b26901d847"}
Nov 02 14:30:22.322417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.322411    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"
Nov 02 14:30:22.325749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.325720    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl" event={"ID":"798e455b-975d-468b-bc10-3e020e992626","Type":"ContainerDied","Data":"e0d225d74e5a8b7eb01e7b7275768d8eec355b553ded8c6286c3cbc8d99f9ba7"}
Nov 02 14:30:22.325905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.325727    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"
Nov 02 14:30:22.328013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.327991    5034 generic.go:356] "Generic (PLEG): container finished" podID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerID="4fb5a15dedbd3cad6d6c9365c6278eef142a4d7fe71efc7f30c6213b320e009b" exitCode=60
Nov 02 14:30:22.328116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.328060    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" event={"ID":"9a96c9cc-4506-402a-8923-620b9381ecc9","Type":"ContainerDied","Data":"4fb5a15dedbd3cad6d6c9365c6278eef142a4d7fe71efc7f30c6213b320e009b"}
Nov 02 14:30:22.328561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.328541    5034 scope.go:122] "RemoveContainer" containerID="4fb5a15dedbd3cad6d6c9365c6278eef142a4d7fe71efc7f30c6213b320e009b"
Nov 02 14:30:22.328922 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:22.328895    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg_odh-ai-gateway-infra(9a96c9cc-4506-402a-8923-620b9381ecc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9"
Nov 02 14:30:22.330027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.329940    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb" event={"ID":"d14778a4-3528-4a2c-b61c-f7ae977edeab","Type":"ContainerDied","Data":"cb76ab978a785bdb89aadc6c1677bd801940c04518fb903e25fb726f31f315a9"}
Nov 02 14:30:22.330027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.330005    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb76ab978a785bdb89aadc6c1677bd801940c04518fb903e25fb726f31f315a9"
Nov 02 14:30:22.330027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.330016    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"
Nov 02 14:30:22.339536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.339382    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"
Nov 02 14:30:22.339536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.340179    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj" event={"ID":"110008e0-3d52-4923-9188-98bafbfccf54","Type":"ContainerDied","Data":"a4ff98dade7571c0572fbc541f627d9ae40611407e2efe00261540a94d26b9a1"}
Nov 02 14:30:22.344182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.344095    5034 scope.go:122] "RemoveContainer" containerID="59ee1b7053593485dfff050075b33777e7e7463aef1b563fc81836df847349f9"
Nov 02 14:30:22.351242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.351215    5034 generic.go:356] "Generic (PLEG): container finished" podID="786d86b7-0109-4fe1-abce-3960750cc80d" containerID="722e5903a60e4a60799100167456d0f981329414663202fa532c15eacaf701e5" exitCode=60
Nov 02 14:30:22.351381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.351361    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" event={"ID":"786d86b7-0109-4fe1-abce-3960750cc80d","Type":"ContainerDied","Data":"722e5903a60e4a60799100167456d0f981329414663202fa532c15eacaf701e5"}
Nov 02 14:30:22.351904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.351880    5034 scope.go:122] "RemoveContainer" containerID="722e5903a60e4a60799100167456d0f981329414663202fa532c15eacaf701e5"
Nov 02 14:30:22.352624 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:22.352304    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6_odh-ai-gateway-infra(786d86b7-0109-4fe1-abce-3960750cc80d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" podUID="786d86b7-0109-4fe1-abce-3960750cc80d"
Nov 02 14:30:22.363303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.363266    5034 scope.go:122] "RemoveContainer" containerID="62cb8fec03b8d2d660b5c7537ad044e37eb4fa182d42b5d8e10a1d4a7217bbd4"
Nov 02 14:30:22.376611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.376583    5034 scope.go:122] "RemoveContainer" containerID="e1b89eff865b5630fdc5b24e140e70ba9d0205aba773d30eba6682d7fcebab09"
Nov 02 14:30:22.388217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.388088    5034 scope.go:122] "RemoveContainer" containerID="416b0020b6c1074a0d0bbaad152311e041a3061fd1f6798eeec40f9b23008b30"
Nov 02 14:30:22.388482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.388459    5034 scope.go:122] "RemoveContainer" containerID="8c161a6bda49a3254c941906822afb21d1a889742a56086537a06ae84603a288"
Nov 02 14:30:22.391322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.391306    5034 scope.go:122] "RemoveContainer" containerID="026b3728a2cf6f7d752de912d9f60c5f606da9bbacd0ff104738a11cc4624db9"
Nov 02 14:30:22.416200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.416034    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"]
Nov 02 14:30:22.416521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.416500    5034 scope.go:122] "RemoveContainer" containerID="4bfcff7ed20416575ba878f60fe6c3e83cacc80d40bd62737cf7b662db3e3393"
Nov 02 14:30:22.423305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.423271    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893830-9gkkl"]
Nov 02 14:30:22.432940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.432541    5034 scope.go:122] "RemoveContainer" containerID="759aff80ff571981c038a04af584cd3d57b2aad37839c1c4e433fce468b94688"
Nov 02 14:30:22.452710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.452684    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"]
Nov 02 14:30:22.523563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.523531    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"]
Nov 02 14:30:22.529663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.529632    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-960a68-29893830-qtwxg"]
Nov 02 14:30:22.555825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.555790    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"]
Nov 02 14:30:22.568063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.568001    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"]
Nov 02 14:30:22.591355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.591318    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a02c35-29893830-lcsdj"]
Nov 02 14:30:22.633814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.633766    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"]
Nov 02 14:30:22.641630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.641576    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893830-7n8qw"]
Nov 02 14:30:22.647509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.647484    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"]
Nov 02 14:30:22.697619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.697583    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"]
Nov 02 14:30:22.798746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.793670    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"]
Nov 02 14:30:22.897078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:22.896719    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"]
Nov 02 14:30:23.010333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.010217    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:23.161791 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.146690    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"]
Nov 02 14:30:23.168566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.168451    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"]
Nov 02 14:30:23.185938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.185900    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7b00a2-30e2-4c1f-bc50-d98158707200-kube-api-access-x265m\" (UniqueName: \"kubernetes.io/projected/6d7b00a2-30e2-4c1f-bc50-d98158707200-kube-api-access-x265m\") pod \"6d7b00a2-30e2-4c1f-bc50-d98158707200\" (UID: \"6d7b00a2-30e2-4c1f-bc50-d98158707200\") "
Nov 02 14:30:23.187338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.186338    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7b00a2-30e2-4c1f-bc50-d98158707200-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b00a2-30e2-4c1f-bc50-d98158707200-maas-api-service-ca\") pod \"6d7b00a2-30e2-4c1f-bc50-d98158707200\" (UID: \"6d7b00a2-30e2-4c1f-bc50-d98158707200\") "
Nov 02 14:30:23.187866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.187626    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:23.188844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.188369    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7b00a2-30e2-4c1f-bc50-d98158707200-maas-api-service-ca" pod "6d7b00a2-30e2-4c1f-bc50-d98158707200" (UID: "6d7b00a2-30e2-4c1f-bc50-d98158707200"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:23.214137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.213969    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7b00a2-30e2-4c1f-bc50-d98158707200-kube-api-access-x265m" pod "6d7b00a2-30e2-4c1f-bc50-d98158707200" (UID: "6d7b00a2-30e2-4c1f-bc50-d98158707200"). InnerVolumeSpecName "kube-api-access-x265m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:23.221282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.221250    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:23.289821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.288565    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5391b42a-6255-4531-957f-31dd8616f975-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5391b42a-6255-4531-957f-31dd8616f975-maas-api-service-ca\") pod \"5391b42a-6255-4531-957f-31dd8616f975\" (UID: \"5391b42a-6255-4531-957f-31dd8616f975\") "
Nov 02 14:30:23.293174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.293137    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5391b42a-6255-4531-957f-31dd8616f975-kube-api-access-qwdnc\" (UniqueName: \"kubernetes.io/projected/5391b42a-6255-4531-957f-31dd8616f975-kube-api-access-qwdnc\") pod \"5391b42a-6255-4531-957f-31dd8616f975\" (UID: \"5391b42a-6255-4531-957f-31dd8616f975\") "
Nov 02 14:30:23.300089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.300051    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d7b00a2-30e2-4c1f-bc50-d98158707200-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:23.300321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.300307    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x265m\" (UniqueName: \"kubernetes.io/projected/6d7b00a2-30e2-4c1f-bc50-d98158707200-kube-api-access-x265m\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:23.300439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.291177    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5391b42a-6255-4531-957f-31dd8616f975-maas-api-service-ca" pod "5391b42a-6255-4531-957f-31dd8616f975" (UID: "5391b42a-6255-4531-957f-31dd8616f975"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:23.319111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.319066    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5391b42a-6255-4531-957f-31dd8616f975-kube-api-access-qwdnc" pod "5391b42a-6255-4531-957f-31dd8616f975" (UID: "5391b42a-6255-4531-957f-31dd8616f975"). InnerVolumeSpecName "kube-api-access-qwdnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:23.365455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.365425    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:23.372507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.372474    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb" event={"ID":"12c10b34-f265-434c-a511-077efe1688d0","Type":"ContainerDied","Data":"2b17cd242825b082418d5eaced9a348545f189e1328fc3ce50c1414a88d72484"}
Nov 02 14:30:23.374753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.374725    5034 scope.go:122] "RemoveContainer" containerID="f4db173ae6b77c6276066ba8d5c11208f78456769bf3258fe75c606bc39455d8"
Nov 02 14:30:23.374968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.374922    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"
Nov 02 14:30:23.384287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.383466    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" event={"ID":"a668bbd8-8a19-42f0-b635-7a88c4cd6b35","Type":"ContainerStarted","Data":"d3163f3830fb1dddeba673410bfe6871e3b6069eff0ff8b79061dcf69d3ac49d"}
Nov 02 14:30:23.390042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.390011    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2" event={"ID":"5391b42a-6255-4531-957f-31dd8616f975","Type":"ContainerDied","Data":"4ba183242cd63eced2ec93e2dd136bcd0642df123b0525680395ff667d2b6580"}
Nov 02 14:30:23.390305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.390290    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"
Nov 02 14:30:23.402727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.401630    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-kube-api-access-ld2cv\" (UniqueName: \"kubernetes.io/projected/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-kube-api-access-ld2cv\") pod \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\" (UID: \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\") "
Nov 02 14:30:23.402727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.401768    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-maas-api-service-ca\") pod \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\" (UID: \"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd\") "
Nov 02 14:30:23.402727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.402490    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5391b42a-6255-4531-957f-31dd8616f975-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:23.402727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.402511    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdnc\" (UniqueName: \"kubernetes.io/projected/5391b42a-6255-4531-957f-31dd8616f975-kube-api-access-qwdnc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:23.405479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.405134    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-maas-api-service-ca" pod "ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" (UID: "ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:23.612823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.567592    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12c10b34-f265-434c-a511-077efe1688d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c10b34-f265-434c-a511-077efe1688d0-maas-api-service-ca\") pod \"12c10b34-f265-434c-a511-077efe1688d0\" (UID: \"12c10b34-f265-434c-a511-077efe1688d0\") "
Nov 02 14:30:23.612823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.567648    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12c10b34-f265-434c-a511-077efe1688d0-kube-api-access-t52w5\" (UniqueName: \"kubernetes.io/projected/12c10b34-f265-434c-a511-077efe1688d0-kube-api-access-t52w5\") pod \"12c10b34-f265-434c-a511-077efe1688d0\" (UID: \"12c10b34-f265-434c-a511-077efe1688d0\") "
Nov 02 14:30:23.612823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.568514    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:23.612823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.572630    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12c10b34-f265-434c-a511-077efe1688d0-maas-api-service-ca" pod "12c10b34-f265-434c-a511-077efe1688d0" (UID: "12c10b34-f265-434c-a511-077efe1688d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:23.687951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.685731    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12c10b34-f265-434c-a511-077efe1688d0-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:23.787830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.783722    5034 scope.go:122] "RemoveContainer" containerID="0204d48376e0ef25e79d88d8d8fbec557c85d9316b15b5dbdd4c0b7493ff7025"
Nov 02 14:30:23.851588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.840015    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12c10b34-f265-434c-a511-077efe1688d0-kube-api-access-t52w5" pod "12c10b34-f265-434c-a511-077efe1688d0" (UID: "12c10b34-f265-434c-a511-077efe1688d0"). InnerVolumeSpecName "kube-api-access-t52w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:23.851588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.840123    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-kube-api-access-ld2cv" pod "ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" (UID: "ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd"). InnerVolumeSpecName "kube-api-access-ld2cv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:23.851588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.847225    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"
Nov 02 14:30:23.861450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.860860    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110008e0-3d52-4923-9188-98bafbfccf54" path="/var/lib/kubelet/pods/110008e0-3d52-4923-9188-98bafbfccf54/volumes"
Nov 02 14:30:23.861562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.861514    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" path="/var/lib/kubelet/pods/1c667d91-1fd4-438c-b3a3-8281cf003b88/volumes"
Nov 02 14:30:23.862151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.862131    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798e455b-975d-468b-bc10-3e020e992626" path="/var/lib/kubelet/pods/798e455b-975d-468b-bc10-3e020e992626/volumes"
Nov 02 14:30:23.863149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.863127    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" path="/var/lib/kubelet/pods/ce138676-ad2e-4761-ad72-571e0b3bf23c/volumes"
Nov 02 14:30:23.865681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.865638    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"]
Nov 02 14:30:23.867245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867221    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerName="cleanup"
Nov 02 14:30:23.867312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867245    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerName="cleanup"
Nov 02 14:30:23.867312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867265    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerName="cleanup"
Nov 02 14:30:23.867312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867272    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerName="cleanup"
Nov 02 14:30:23.867312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867299    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerName="cleanup"
Nov 02 14:30:23.867312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867309    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867338    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5391b42a-6255-4531-957f-31dd8616f975" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867346    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5391b42a-6255-4531-957f-31dd8616f975" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867375    5034 cpu_manager.go:414] "containerMap: removing container" podUID="12c10b34-f265-434c-a511-077efe1688d0" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867382    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c10b34-f265-434c-a511-077efe1688d0" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867392    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867399    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867414    5034 cpu_manager.go:414] "containerMap: removing container" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867421    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867460    5034 cpu_manager.go:414] "containerMap: removing container" podUID="798e455b-975d-468b-bc10-3e020e992626" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867467    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798e455b-975d-468b-bc10-3e020e992626" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867478    5034 cpu_manager.go:414] "containerMap: removing container" podUID="110008e0-3d52-4923-9188-98bafbfccf54" containerName="cleanup"
Nov 02 14:30:23.867491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867484    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110008e0-3d52-4923-9188-98bafbfccf54" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867496    5034 cpu_manager.go:414] "containerMap: removing container" podUID="110008e0-3d52-4923-9188-98bafbfccf54" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867503    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110008e0-3d52-4923-9188-98bafbfccf54" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867536    5034 cpu_manager.go:414] "containerMap: removing container" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867544    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867560    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867567    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867578    5034 cpu_manager.go:414] "containerMap: removing container" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867586    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867679    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867687    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867700    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867706    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867722    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867729    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867738    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867745    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867757    5034 cpu_manager.go:414] "containerMap: removing container" podUID="110008e0-3d52-4923-9188-98bafbfccf54" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867763    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110008e0-3d52-4923-9188-98bafbfccf54" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867803    5034 cpu_manager.go:414] "containerMap: removing container" podUID="12c10b34-f265-434c-a511-077efe1688d0" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867810    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c10b34-f265-434c-a511-077efe1688d0" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867826    5034 cpu_manager.go:414] "containerMap: removing container" podUID="798e455b-975d-468b-bc10-3e020e992626" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867832    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798e455b-975d-468b-bc10-3e020e992626" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867846    5034 cpu_manager.go:414] "containerMap: removing container" podUID="798e455b-975d-468b-bc10-3e020e992626" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867853    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798e455b-975d-468b-bc10-3e020e992626" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867867    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867874    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867884    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867890    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867900    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d14778a4-3528-4a2c-b61c-f7ae977edeab" containerName="revoke-keys"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867907    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d14778a4-3528-4a2c-b61c-f7ae977edeab" containerName="revoke-keys"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867920    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5391b42a-6255-4531-957f-31dd8616f975" containerName="cleanup"
Nov 02 14:30:23.868004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.867926    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5391b42a-6255-4531-957f-31dd8616f975" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868151    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868166    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="798e455b-975d-468b-bc10-3e020e992626" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868254    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="110008e0-3d52-4923-9188-98bafbfccf54" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868270    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868279    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="110008e0-3d52-4923-9188-98bafbfccf54" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868289    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c10b34-f265-434c-a511-077efe1688d0" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868298    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868309    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868319    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868328    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868337    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="110008e0-3d52-4923-9188-98bafbfccf54" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868345    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868354    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868364    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868393    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868403    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="798e455b-975d-468b-bc10-3e020e992626" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868414    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868423    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868435    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868445    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5391b42a-6255-4531-957f-31dd8616f975" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868452    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d14778a4-3528-4a2c-b61c-f7ae977edeab" containerName="revoke-keys"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868462    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5391b42a-6255-4531-957f-31dd8616f975" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868477    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868485    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerName="cleanup"
Nov 02 14:30:23.869160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.868497    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c10b34-f265-434c-a511-077efe1688d0" containerName="cleanup"
Nov 02 14:30:23.884326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.884230    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"]
Nov 02 14:30:23.884326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.884278    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7" event={"ID":"6d7b00a2-30e2-4c1f-bc50-d98158707200","Type":"ContainerDied","Data":"5ecade2a1f02c5f3b9ea0387c772af9656a402198dc2afbb37eca09d11b0e871"}
Nov 02 14:30:23.884547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.884392    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:23.892817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.891292    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld2cv\" (UniqueName: \"kubernetes.io/projected/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd-kube-api-access-ld2cv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:23.892817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.891320    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t52w5\" (UniqueName: \"kubernetes.io/projected/12c10b34-f265-434c-a511-077efe1688d0-kube-api-access-t52w5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:23.908842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.906297    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:30:23.908842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.907300    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" event={"ID":"5267a8a7-bd52-4e1b-87eb-d6c4085e0769","Type":"ContainerStarted","Data":"2a4ee1086ef70473efcef6e0de44b39627cba95086c320b518d4323df4bc1e9d"}
Nov 02 14:30:23.908842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.907929    5034 scope.go:122] "RemoveContainer" containerID="2a4ee1086ef70473efcef6e0de44b39627cba95086c320b518d4323df4bc1e9d"
Nov 02 14:30:23.908842 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:23.908363    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb_odh-ai-gateway-infra(5267a8a7-bd52-4e1b-87eb-d6c4085e0769)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769"
Nov 02 14:30:23.920050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.919512    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4" event={"ID":"ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd","Type":"ContainerDied","Data":"d2218203fe008bef1fa8fca7ebd3141060d7b5fdc372a72cb99cc171c048a038"}
Nov 02 14:30:23.920050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.919624    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"
Nov 02 14:30:23.935383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.929029    5034 scope.go:122] "RemoveContainer" containerID="e0d059b546bf734ef1632ca024d7f837a876d306497e9ddd231d584443663b94"
Nov 02 14:30:23.939302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.938329    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" podStartSLOduration=23.938311605 podStartE2EDuration="23.938311605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:23.934893945 +0000 UTC m=+2814.686198384" watchObservedRunningTime="2026-11-02 14:30:23.938311605 +0000 UTC m=+2814.689616046"
Nov 02 14:30:23.969410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.969390    5034 scope.go:122] "RemoveContainer" containerID="861b49bfffa13cdbf4f95990ba6bac26962029d821926309617b2f3a12b3a6c2"
Nov 02 14:30:23.978201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.978116    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:23.999050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:23.999003    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad4c0002-wmrl2\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.003139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.003100    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvh5c\" (UniqueName: \"kubernetes.io/projected/904a4b88-327f-4bfc-bf1e-28d0ad054106-kube-api-access-dvh5c\") pod \"maas-api-revoke-sub-ad4c0002-wmrl2\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.003523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.003505    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad4c0002-wmrl2\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.097339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.097136    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:24.107887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.107854    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-kube-api-access-9wvwm\" (UniqueName: \"kubernetes.io/projected/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-kube-api-access-9wvwm\") pod \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\" (UID: \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\") "
Nov 02 14:30:24.108324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.108302    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-maas-api-service-ca\") pod \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\" (UID: \"1c6427b3-906d-47bc-938f-c38f0c1b5b5a\") "
Nov 02 14:30:24.108842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.108816    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad4c0002-wmrl2\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.109151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.109129    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad4c0002-wmrl2\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.109493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.109474    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvh5c\" (UniqueName: \"kubernetes.io/projected/904a4b88-327f-4bfc-bf1e-28d0ad054106-kube-api-access-dvh5c\") pod \"maas-api-revoke-sub-ad4c0002-wmrl2\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.110841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.110819    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad4c0002-wmrl2\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.113859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.113833    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-kube-api-access-9wvwm" pod "1c6427b3-906d-47bc-938f-c38f0c1b5b5a" (UID: "1c6427b3-906d-47bc-938f-c38f0c1b5b5a"). InnerVolumeSpecName "kube-api-access-9wvwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:24.118429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.118362    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad4c0002-wmrl2\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.118938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.118915    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-maas-api-service-ca" pod "1c6427b3-906d-47bc-938f-c38f0c1b5b5a" (UID: "1c6427b3-906d-47bc-938f-c38f0c1b5b5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:24.135341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.135303    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvh5c\" (UniqueName: \"kubernetes.io/projected/904a4b88-327f-4bfc-bf1e-28d0ad054106-kube-api-access-dvh5c\") pod \"maas-api-revoke-sub-ad4c0002-wmrl2\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.168251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.168209    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:24.196145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.196118    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:24.210666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.210627    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732c703d-dd22-4a0b-a2c0-93e2ac208979-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c703d-dd22-4a0b-a2c0-93e2ac208979-maas-api-service-ca\") pod \"732c703d-dd22-4a0b-a2c0-93e2ac208979\" (UID: \"732c703d-dd22-4a0b-a2c0-93e2ac208979\") "
Nov 02 14:30:24.210854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.210726    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732c703d-dd22-4a0b-a2c0-93e2ac208979-kube-api-access-qmpdq\" (UniqueName: \"kubernetes.io/projected/732c703d-dd22-4a0b-a2c0-93e2ac208979-kube-api-access-qmpdq\") pod \"732c703d-dd22-4a0b-a2c0-93e2ac208979\" (UID: \"732c703d-dd22-4a0b-a2c0-93e2ac208979\") "
Nov 02 14:30:24.211142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.211110    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732c703d-dd22-4a0b-a2c0-93e2ac208979-maas-api-service-ca" pod "732c703d-dd22-4a0b-a2c0-93e2ac208979" (UID: "732c703d-dd22-4a0b-a2c0-93e2ac208979"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:24.211477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.211451    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:24.211578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.211480    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvwm\" (UniqueName: \"kubernetes.io/projected/1c6427b3-906d-47bc-938f-c38f0c1b5b5a-kube-api-access-9wvwm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:24.211578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.211495    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732c703d-dd22-4a0b-a2c0-93e2ac208979-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:24.213096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.213069    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732c703d-dd22-4a0b-a2c0-93e2ac208979-kube-api-access-qmpdq" pod "732c703d-dd22-4a0b-a2c0-93e2ac208979" (UID: "732c703d-dd22-4a0b-a2c0-93e2ac208979"). InnerVolumeSpecName "kube-api-access-qmpdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:24.258806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.258742    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"]
Nov 02 14:30:24.264372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.264338    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893830-qp5c7"]
Nov 02 14:30:24.286960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.286910    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:24.301109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.301072    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"]
Nov 02 14:30:24.307395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.307364    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893830-xvmf2"]
Nov 02 14:30:24.312840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.312809    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a96c9cc-4506-402a-8923-620b9381ecc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a96c9cc-4506-402a-8923-620b9381ecc9-maas-api-service-ca\") pod \"9a96c9cc-4506-402a-8923-620b9381ecc9\" (UID: \"9a96c9cc-4506-402a-8923-620b9381ecc9\") "
Nov 02 14:30:24.312986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.312941    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786d86b7-0109-4fe1-abce-3960750cc80d-kube-api-access-v6tmd\" (UniqueName: \"kubernetes.io/projected/786d86b7-0109-4fe1-abce-3960750cc80d-kube-api-access-v6tmd\") pod \"786d86b7-0109-4fe1-abce-3960750cc80d\" (UID: \"786d86b7-0109-4fe1-abce-3960750cc80d\") "
Nov 02 14:30:24.313772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.313177    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a96c9cc-4506-402a-8923-620b9381ecc9-kube-api-access-s2j2h\" (UniqueName: \"kubernetes.io/projected/9a96c9cc-4506-402a-8923-620b9381ecc9-kube-api-access-s2j2h\") pod \"9a96c9cc-4506-402a-8923-620b9381ecc9\" (UID: \"9a96c9cc-4506-402a-8923-620b9381ecc9\") "
Nov 02 14:30:24.313772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.313247    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a96c9cc-4506-402a-8923-620b9381ecc9-maas-api-service-ca" pod "9a96c9cc-4506-402a-8923-620b9381ecc9" (UID: "9a96c9cc-4506-402a-8923-620b9381ecc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:24.313772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.313314    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786d86b7-0109-4fe1-abce-3960750cc80d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786d86b7-0109-4fe1-abce-3960750cc80d-maas-api-service-ca\") pod \"786d86b7-0109-4fe1-abce-3960750cc80d\" (UID: \"786d86b7-0109-4fe1-abce-3960750cc80d\") "
Nov 02 14:30:24.313772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.313724    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786d86b7-0109-4fe1-abce-3960750cc80d-maas-api-service-ca" pod "786d86b7-0109-4fe1-abce-3960750cc80d" (UID: "786d86b7-0109-4fe1-abce-3960750cc80d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:24.314251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.314222    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/786d86b7-0109-4fe1-abce-3960750cc80d-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:24.314336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.314255    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmpdq\" (UniqueName: \"kubernetes.io/projected/732c703d-dd22-4a0b-a2c0-93e2ac208979-kube-api-access-qmpdq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:24.314336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.314269    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a96c9cc-4506-402a-8923-620b9381ecc9-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:24.315855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.315823    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786d86b7-0109-4fe1-abce-3960750cc80d-kube-api-access-v6tmd" pod "786d86b7-0109-4fe1-abce-3960750cc80d" (UID: "786d86b7-0109-4fe1-abce-3960750cc80d"). InnerVolumeSpecName "kube-api-access-v6tmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:24.315947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.315912    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a96c9cc-4506-402a-8923-620b9381ecc9-kube-api-access-s2j2h" pod "9a96c9cc-4506-402a-8923-620b9381ecc9" (UID: "9a96c9cc-4506-402a-8923-620b9381ecc9"). InnerVolumeSpecName "kube-api-access-s2j2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:24.359157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.358799    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"]
Nov 02 14:30:24.366576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.366546    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893830-nfrb4"]
Nov 02 14:30:24.417337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.417289    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2j2h\" (UniqueName: \"kubernetes.io/projected/9a96c9cc-4506-402a-8923-620b9381ecc9-kube-api-access-s2j2h\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:24.417574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.417550    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6tmd\" (UniqueName: \"kubernetes.io/projected/786d86b7-0109-4fe1-abce-3960750cc80d-kube-api-access-v6tmd\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:24.435395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.435348    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"]
Nov 02 14:30:24.441465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.441427    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-5f98a6-29893830-kdxdb"]
Nov 02 14:30:24.611721 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:24.611676    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904a4b88_327f_4bfc_bf1e_28d0ad054106.slice/crio-c52a25ed7c30a5ec78a0680f058bba5736a2085ebbd85ba6b16c542a5d723bcc WatchSource:0}: Error finding container c52a25ed7c30a5ec78a0680f058bba5736a2085ebbd85ba6b16c542a5d723bcc: Status 404 returned error can't find the container with id c52a25ed7c30a5ec78a0680f058bba5736a2085ebbd85ba6b16c542a5d723bcc
Nov 02 14:30:24.613343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.613313    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"]
Nov 02 14:30:24.932351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.932320    5034 generic.go:356] "Generic (PLEG): container finished" podID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerID="2a4ee1086ef70473efcef6e0de44b39627cba95086c320b518d4323df4bc1e9d" exitCode=60
Nov 02 14:30:24.932522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.932400    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" event={"ID":"5267a8a7-bd52-4e1b-87eb-d6c4085e0769","Type":"ContainerDied","Data":"2a4ee1086ef70473efcef6e0de44b39627cba95086c320b518d4323df4bc1e9d"}
Nov 02 14:30:24.932522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.932477    5034 scope.go:122] "RemoveContainer" containerID="416b0020b6c1074a0d0bbaad152311e041a3061fd1f6798eeec40f9b23008b30"
Nov 02 14:30:24.932983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.932959    5034 scope.go:122] "RemoveContainer" containerID="2a4ee1086ef70473efcef6e0de44b39627cba95086c320b518d4323df4bc1e9d"
Nov 02 14:30:24.933571 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:24.933542    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb_odh-ai-gateway-infra(5267a8a7-bd52-4e1b-87eb-d6c4085e0769)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769"
Nov 02 14:30:24.934222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.934204    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"
Nov 02 14:30:24.934327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.934212    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6" event={"ID":"786d86b7-0109-4fe1-abce-3960750cc80d","Type":"ContainerDied","Data":"067a67f9edb01ffff5f5cfe3937c67ea9e1bf092e08e2185fc7e8e8395a1b25e"}
Nov 02 14:30:24.937539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.937516    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"
Nov 02 14:30:24.937539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.937527    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd" event={"ID":"1c6427b3-906d-47bc-938f-c38f0c1b5b5a","Type":"ContainerDied","Data":"90fbb95cbba1de89194c7f1f021da577be8a27b92dff26207750632f5646466c"}
Nov 02 14:30:24.939322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.939290    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2" event={"ID":"904a4b88-327f-4bfc-bf1e-28d0ad054106","Type":"ContainerStarted","Data":"c52a25ed7c30a5ec78a0680f058bba5736a2085ebbd85ba6b16c542a5d723bcc"}
Nov 02 14:30:24.945069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.945048    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"
Nov 02 14:30:24.945201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.945108    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9" event={"ID":"732c703d-dd22-4a0b-a2c0-93e2ac208979","Type":"ContainerDied","Data":"4360ed860c9654956784e37dede48af7574639c7e57e5156ec277e047c31626e"}
Nov 02 14:30:24.948905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.948875    5034 generic.go:356] "Generic (PLEG): container finished" podID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerID="d3163f3830fb1dddeba673410bfe6871e3b6069eff0ff8b79061dcf69d3ac49d" exitCode=60
Nov 02 14:30:24.949155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.949136    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" event={"ID":"a668bbd8-8a19-42f0-b635-7a88c4cd6b35","Type":"ContainerDied","Data":"d3163f3830fb1dddeba673410bfe6871e3b6069eff0ff8b79061dcf69d3ac49d"}
Nov 02 14:30:24.950974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.949812    5034 scope.go:122] "RemoveContainer" containerID="d3163f3830fb1dddeba673410bfe6871e3b6069eff0ff8b79061dcf69d3ac49d"
Nov 02 14:30:24.950974 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:24.950206    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c_odh-ai-gateway-infra(a668bbd8-8a19-42f0-b635-7a88c4cd6b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35"
Nov 02 14:30:24.952714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.952679    5034 scope.go:122] "RemoveContainer" containerID="722e5903a60e4a60799100167456d0f981329414663202fa532c15eacaf701e5"
Nov 02 14:30:24.955980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.955056    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg" event={"ID":"9a96c9cc-4506-402a-8923-620b9381ecc9","Type":"ContainerDied","Data":"77ad31de9317bb8f723f0bb1c6e61974058dce5f4d8d62b0a1ac4236a68321bb"}
Nov 02 14:30:24.955980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.955065    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"
Nov 02 14:30:24.955980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.955359    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"]
Nov 02 14:30:24.968373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.968281    5034 scope.go:122] "RemoveContainer" containerID="ef8cff6d45454de5012bf5a031232a3f881d7db2239b14df89afbef68827d497"
Nov 02 14:30:24.984812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.984754    5034 scope.go:122] "RemoveContainer" containerID="0f2927942e35af93cbe2144b1ce5af5bf38b5f0dc7d6e77b86c521964ee5ec68"
Nov 02 14:30:24.999969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:24.999944    5034 scope.go:122] "RemoveContainer" containerID="8c161a6bda49a3254c941906822afb21d1a889742a56086537a06ae84603a288"
Nov 02 14:30:25.016703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.016678    5034 scope.go:122] "RemoveContainer" containerID="4fb5a15dedbd3cad6d6c9365c6278eef142a4d7fe71efc7f30c6213b320e009b"
Nov 02 14:30:25.048746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.048708    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"]
Nov 02 14:30:25.054039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.054001    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893830-z6psd"]
Nov 02 14:30:25.083977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.083945    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"]
Nov 02 14:30:25.105399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.105367    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"]
Nov 02 14:30:25.139384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.139278    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893830-nmpx6"]
Nov 02 14:30:25.209642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.209604    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"]
Nov 02 14:30:25.215509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.215476    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1a562c-29893830-rr9xg"]
Nov 02 14:30:25.250751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.250718    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"]
Nov 02 14:30:25.256476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.256443    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893830-lkbc9"]
Nov 02 14:30:25.399989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.399894    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12c10b34-f265-434c-a511-077efe1688d0" path="/var/lib/kubelet/pods/12c10b34-f265-434c-a511-077efe1688d0/volumes"
Nov 02 14:30:25.400602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.400581    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" path="/var/lib/kubelet/pods/1c6427b3-906d-47bc-938f-c38f0c1b5b5a/volumes"
Nov 02 14:30:25.401210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.401191    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5391b42a-6255-4531-957f-31dd8616f975" path="/var/lib/kubelet/pods/5391b42a-6255-4531-957f-31dd8616f975/volumes"
Nov 02 14:30:25.402225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.402204    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" path="/var/lib/kubelet/pods/6d7b00a2-30e2-4c1f-bc50-d98158707200/volumes"
Nov 02 14:30:25.402805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.402770    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" path="/var/lib/kubelet/pods/732c703d-dd22-4a0b-a2c0-93e2ac208979/volumes"
Nov 02 14:30:25.403382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.403364    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" path="/var/lib/kubelet/pods/786d86b7-0109-4fe1-abce-3960750cc80d/volumes"
Nov 02 14:30:25.404346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.404321    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" path="/var/lib/kubelet/pods/9a96c9cc-4506-402a-8923-620b9381ecc9/volumes"
Nov 02 14:30:25.404921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.404902    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" path="/var/lib/kubelet/pods/ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd/volumes"
Nov 02 14:30:25.967748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.967710    5034 generic.go:356] "Generic (PLEG): container finished" podID="904a4b88-327f-4bfc-bf1e-28d0ad054106" containerID="0db51c5a9c61b3ef323a84ec3aee022b96b4f37f60d62a9cd3dd1531a2a1fbf0" exitCode=0
Nov 02 14:30:25.967939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:25.967822    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2" event={"ID":"904a4b88-327f-4bfc-bf1e-28d0ad054106","Type":"ContainerDied","Data":"0db51c5a9c61b3ef323a84ec3aee022b96b4f37f60d62a9cd3dd1531a2a1fbf0"}
Nov 02 14:30:26.207357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.207328    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:26.252460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.252432    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:26.357273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.357241    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-maas-api-service-ca\") pod \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\" (UID: \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\") "
Nov 02 14:30:26.357453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.357311    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-maas-api-service-ca\") pod \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\" (UID: \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\") "
Nov 02 14:30:26.357453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.357337    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-kube-api-access-lxbcf\") pod \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\" (UID: \"a668bbd8-8a19-42f0-b635-7a88c4cd6b35\") "
Nov 02 14:30:26.357602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.357463    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-kube-api-access-ljrt2\" (UniqueName: \"kubernetes.io/projected/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-kube-api-access-ljrt2\") pod \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\" (UID: \"5267a8a7-bd52-4e1b-87eb-d6c4085e0769\") "
Nov 02 14:30:26.357673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.357594    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-maas-api-service-ca" pod "a668bbd8-8a19-42f0-b635-7a88c4cd6b35" (UID: "a668bbd8-8a19-42f0-b635-7a88c4cd6b35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:26.357734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.357661    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-maas-api-service-ca" pod "5267a8a7-bd52-4e1b-87eb-d6c4085e0769" (UID: "5267a8a7-bd52-4e1b-87eb-d6c4085e0769"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:26.358387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.358364    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:26.358387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.358387    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:26.359837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.359815    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-kube-api-access-ljrt2" pod "5267a8a7-bd52-4e1b-87eb-d6c4085e0769" (UID: "5267a8a7-bd52-4e1b-87eb-d6c4085e0769"). InnerVolumeSpecName "kube-api-access-ljrt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:26.359928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.359844    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-kube-api-access-lxbcf" pod "a668bbd8-8a19-42f0-b635-7a88c4cd6b35" (UID: "a668bbd8-8a19-42f0-b635-7a88c4cd6b35"). InnerVolumeSpecName "kube-api-access-lxbcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:26.460016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.459924    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljrt2\" (UniqueName: \"kubernetes.io/projected/5267a8a7-bd52-4e1b-87eb-d6c4085e0769-kube-api-access-ljrt2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:26.460016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.459956    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbcf\" (UniqueName: \"kubernetes.io/projected/a668bbd8-8a19-42f0-b635-7a88c4cd6b35-kube-api-access-lxbcf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:26.987488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.987448    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"
Nov 02 14:30:26.987488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.987467    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb" event={"ID":"5267a8a7-bd52-4e1b-87eb-d6c4085e0769","Type":"ContainerDied","Data":"92f610a6eade6b1f9f9d738dcda9cb384a02645fe760fc93e9607887d3502428"}
Nov 02 14:30:26.987731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.987545    5034 scope.go:122] "RemoveContainer" containerID="2a4ee1086ef70473efcef6e0de44b39627cba95086c320b518d4323df4bc1e9d"
Nov 02 14:30:26.989520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.989495    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"
Nov 02 14:30:26.989674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:26.989649    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c" event={"ID":"a668bbd8-8a19-42f0-b635-7a88c4cd6b35","Type":"ContainerDied","Data":"deb9911ddfd072a63803a852093b49af0cc70f870e1df0bd5a340e26259d64a0"}
Nov 02 14:30:27.005460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.005131    5034 scope.go:122] "RemoveContainer" containerID="d3163f3830fb1dddeba673410bfe6871e3b6069eff0ff8b79061dcf69d3ac49d"
Nov 02 14:30:27.068310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.067426    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"]
Nov 02 14:30:27.087827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.082283    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893830-lxkzb"]
Nov 02 14:30:27.111904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.111857    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"]
Nov 02 14:30:27.120405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.120156    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-26b508-29893830-qbb8c"]
Nov 02 14:30:27.186722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.186691    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:27.275838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.275772    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-openshift-ca\") pod \"904a4b88-327f-4bfc-bf1e-28d0ad054106\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") "
Nov 02 14:30:27.276061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.275858    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-xks-ca\") pod \"904a4b88-327f-4bfc-bf1e-28d0ad054106\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") "
Nov 02 14:30:27.276061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.275980    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904a4b88-327f-4bfc-bf1e-28d0ad054106-kube-api-access-dvh5c\" (UniqueName: \"kubernetes.io/projected/904a4b88-327f-4bfc-bf1e-28d0ad054106-kube-api-access-dvh5c\") pod \"904a4b88-327f-4bfc-bf1e-28d0ad054106\" (UID: \"904a4b88-327f-4bfc-bf1e-28d0ad054106\") "
Nov 02 14:30:27.276604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.276577    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-openshift-ca" pod "904a4b88-327f-4bfc-bf1e-28d0ad054106" (UID: "904a4b88-327f-4bfc-bf1e-28d0ad054106"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:27.277041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.277015    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:27.279351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.279315    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904a4b88-327f-4bfc-bf1e-28d0ad054106-kube-api-access-dvh5c" pod "904a4b88-327f-4bfc-bf1e-28d0ad054106" (UID: "904a4b88-327f-4bfc-bf1e-28d0ad054106"). InnerVolumeSpecName "kube-api-access-dvh5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:27.279494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.279473    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-xks-ca" pod "904a4b88-327f-4bfc-bf1e-28d0ad054106" (UID: "904a4b88-327f-4bfc-bf1e-28d0ad054106"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:30:27.379094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.379055    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904a4b88-327f-4bfc-bf1e-28d0ad054106-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:27.379094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.379091    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvh5c\" (UniqueName: \"kubernetes.io/projected/904a4b88-327f-4bfc-bf1e-28d0ad054106-kube-api-access-dvh5c\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:27.403077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.402987    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" path="/var/lib/kubelet/pods/5267a8a7-bd52-4e1b-87eb-d6c4085e0769/volumes"
Nov 02 14:30:27.403652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:27.403630    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" path="/var/lib/kubelet/pods/a668bbd8-8a19-42f0-b635-7a88c4cd6b35/volumes"
Nov 02 14:30:28.002788 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:28.002741    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"
Nov 02 14:30:28.003290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:28.002763    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2" event={"ID":"904a4b88-327f-4bfc-bf1e-28d0ad054106","Type":"ContainerDied","Data":"c52a25ed7c30a5ec78a0680f058bba5736a2085ebbd85ba6b16c542a5d723bcc"}
Nov 02 14:30:28.003290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:28.002844    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c52a25ed7c30a5ec78a0680f058bba5736a2085ebbd85ba6b16c542a5d723bcc"
Nov 02 14:30:34.522740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.522700    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"]
Nov 02 14:30:34.523577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523557    5034 cpu_manager.go:414] "containerMap: removing container" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" containerName="cleanup"
Nov 02 14:30:34.523577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523575    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523595    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523601    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523609    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523615    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523628    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523633    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523639    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523643    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c667d91-1fd4-438c-b3a3-8281cf003b88" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523652    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523656    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523664    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523669    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523675    5034 cpu_manager.go:414] "containerMap: removing container" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523680    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523686    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523690    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523700    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523704    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523718    5034 cpu_manager.go:414] "containerMap: removing container" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523723    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523732    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523738    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523747    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523751    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7b00a2-30e2-4c1f-bc50-d98158707200" containerName="cleanup"
Nov 02 14:30:34.523755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523772    5034 cpu_manager.go:414] "containerMap: removing container" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523790    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523797    5034 cpu_manager.go:414] "containerMap: removing container" podUID="904a4b88-327f-4bfc-bf1e-28d0ad054106" containerName="revoke-keys"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523802    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904a4b88-327f-4bfc-bf1e-28d0ad054106" containerName="revoke-keys"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523812    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523816    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523823    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523828    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1bf66ee-0739-4042-9f1d-d1f0f5de7f68" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523836    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523840    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523846    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523850    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523860    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5391b42a-6255-4531-957f-31dd8616f975" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523864    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5391b42a-6255-4531-957f-31dd8616f975" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523870    5034 cpu_manager.go:414] "containerMap: removing container" podUID="12c10b34-f265-434c-a511-077efe1688d0" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.523874    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12c10b34-f265-434c-a511-077efe1688d0" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524004    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f54dd3-b21f-4705-af6f-d29c35b9368c" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524014    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524020    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524025    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524031    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524037    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce138676-ad2e-4761-ad72-571e0b3bf23c" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524045    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524052    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524058    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524064    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524070    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="798e455b-975d-468b-bc10-3e020e992626" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524076    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8b3f79-d31a-4dc5-bb5a-0aaae7dc8dfd" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524083    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524089    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5391b42a-6255-4531-957f-31dd8616f975" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524099    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524105    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524110    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="904a4b88-327f-4bfc-bf1e-28d0ad054106" containerName="revoke-keys"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524116    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524128    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="12c10b34-f265-434c-a511-077efe1688d0" containerName="cleanup"
Nov 02 14:30:34.525087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.524135    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerName="cleanup"
Nov 02 14:30:34.529576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.529555    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.557927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.557877    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"]
Nov 02 14:30:34.568158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.568122    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:30:34.568158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.568122    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:30:34.568411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.568211    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:30:34.568528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.568500    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:30:34.648961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.648917    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcwmk\" (UniqueName: \"kubernetes.io/projected/71b6bd36-75d9-4a4e-a569-f1d181b84c13-kube-api-access-kcwmk\") pod \"maas-api-revoke-sub-24ef0910-j489g\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.649204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.649162    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ef0910-j489g\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.649204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.649191    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ef0910-j489g\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.750541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.750502    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ef0910-j489g\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.750541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.750541    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ef0910-j489g\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.751053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.750804    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcwmk\" (UniqueName: \"kubernetes.io/projected/71b6bd36-75d9-4a4e-a569-f1d181b84c13-kube-api-access-kcwmk\") pod \"maas-api-revoke-sub-24ef0910-j489g\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.751312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.751283    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24ef0910-j489g\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.753212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.753190    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24ef0910-j489g\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.776195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.776127    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcwmk\" (UniqueName: \"kubernetes.io/projected/71b6bd36-75d9-4a4e-a569-f1d181b84c13-kube-api-access-kcwmk\") pod \"maas-api-revoke-sub-24ef0910-j489g\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:34.853309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:34.853270    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:35.168938 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:35.168898    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b6bd36_75d9_4a4e_a569_f1d181b84c13.slice/crio-1b4fee41a7813f3973ec3021c80a7cfd83afa57724814c3d92ab019b14daab2e WatchSource:0}: Error finding container 1b4fee41a7813f3973ec3021c80a7cfd83afa57724814c3d92ab019b14daab2e: Status 404 returned error can't find the container with id 1b4fee41a7813f3973ec3021c80a7cfd83afa57724814c3d92ab019b14daab2e
Nov 02 14:30:35.171654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.171626    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"]
Nov 02 14:30:35.416364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.416282    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"]
Nov 02 14:30:35.417207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417188    5034 cpu_manager.go:414] "containerMap: removing container" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" containerName="cleanup"
Nov 02 14:30:35.417207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417207    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" containerName="cleanup"
Nov 02 14:30:35.417327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417228    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerName="cleanup"
Nov 02 14:30:35.417327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417234    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerName="cleanup"
Nov 02 14:30:35.417327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417268    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerName="cleanup"
Nov 02 14:30:35.417327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417272    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerName="cleanup"
Nov 02 14:30:35.417327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417288    5034 cpu_manager.go:414] "containerMap: removing container" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerName="cleanup"
Nov 02 14:30:35.417327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417294    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732c703d-dd22-4a0b-a2c0-93e2ac208979" containerName="cleanup"
Nov 02 14:30:35.417527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417417    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a96c9cc-4506-402a-8923-620b9381ecc9" containerName="cleanup"
Nov 02 14:30:35.417527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417427    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5267a8a7-bd52-4e1b-87eb-d6c4085e0769" containerName="cleanup"
Nov 02 14:30:35.417527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417436    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c6427b3-906d-47bc-938f-c38f0c1b5b5a" containerName="cleanup"
Nov 02 14:30:35.417527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417444    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a668bbd8-8a19-42f0-b635-7a88c4cd6b35" containerName="cleanup"
Nov 02 14:30:35.417527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.417452    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="786d86b7-0109-4fe1-abce-3960750cc80d" containerName="cleanup"
Nov 02 14:30:35.425359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.425323    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.448032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.447992    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"]
Nov 02 14:30:35.481244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.481203    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e0b5385-72jrx\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.481436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.481346    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e0b5385-72jrx\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.481744 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.481724    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62qht\" (UniqueName: \"kubernetes.io/projected/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-kube-api-access-62qht\") pod \"maas-api-revoke-sub-2e0b5385-72jrx\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.583010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.582971    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e0b5385-72jrx\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.583467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.583212    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62qht\" (UniqueName: \"kubernetes.io/projected/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-kube-api-access-62qht\") pod \"maas-api-revoke-sub-2e0b5385-72jrx\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.583467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.583307    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e0b5385-72jrx\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.583696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.583673    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e0b5385-72jrx\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.586050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.586024    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e0b5385-72jrx\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.608929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.608891    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62qht\" (UniqueName: \"kubernetes.io/projected/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-kube-api-access-62qht\") pod \"maas-api-revoke-sub-2e0b5385-72jrx\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:35.751464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:35.751433    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:36.078832 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:36.076542    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d6b85e_f4cf_4ec3_aab9_d43d532c42f3.slice/crio-f6bb6562bbc3d8589976027e8de708087eb277163cfcce42dc6b5f11ef54adb9 WatchSource:0}: Error finding container f6bb6562bbc3d8589976027e8de708087eb277163cfcce42dc6b5f11ef54adb9: Status 404 returned error can't find the container with id f6bb6562bbc3d8589976027e8de708087eb277163cfcce42dc6b5f11ef54adb9
Nov 02 14:30:36.079498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:36.079376    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"]
Nov 02 14:30:36.165296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:36.165259    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g" event={"ID":"71b6bd36-75d9-4a4e-a569-f1d181b84c13","Type":"ContainerStarted","Data":"d16306f483b8561e7fc25a35c82d9bbfe505a16b3d86621444d11e90f901fbc9"}
Nov 02 14:30:36.165481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:36.165317    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g" event={"ID":"71b6bd36-75d9-4a4e-a569-f1d181b84c13","Type":"ContainerStarted","Data":"1b4fee41a7813f3973ec3021c80a7cfd83afa57724814c3d92ab019b14daab2e"}
Nov 02 14:30:36.166805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:36.166761    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx" event={"ID":"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3","Type":"ContainerStarted","Data":"f6bb6562bbc3d8589976027e8de708087eb277163cfcce42dc6b5f11ef54adb9"}
Nov 02 14:30:36.211082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:36.211000    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g" podStartSLOduration=2.21098175 podStartE2EDuration="2.21098175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:36.209723994 +0000 UTC m=+2826.961028431" watchObservedRunningTime="2026-11-02 14:30:36.21098175 +0000 UTC m=+2826.962286189"
Nov 02 14:30:37.179813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:37.179763    5034 generic.go:356] "Generic (PLEG): container finished" podID="46d6b85e-f4cf-4ec3-aab9-d43d532c42f3" containerID="e484945003a838940d4ffed7e6c386088a603420c71f8fd161e37e20849134ae" exitCode=0
Nov 02 14:30:37.180335 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:37.179847    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx" event={"ID":"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3","Type":"ContainerDied","Data":"e484945003a838940d4ffed7e6c386088a603420c71f8fd161e37e20849134ae"}
Nov 02 14:30:37.181969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:37.181940    5034 generic.go:356] "Generic (PLEG): container finished" podID="71b6bd36-75d9-4a4e-a569-f1d181b84c13" containerID="d16306f483b8561e7fc25a35c82d9bbfe505a16b3d86621444d11e90f901fbc9" exitCode=0
Nov 02 14:30:37.182093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:37.182022    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g" event={"ID":"71b6bd36-75d9-4a4e-a569-f1d181b84c13","Type":"ContainerDied","Data":"d16306f483b8561e7fc25a35c82d9bbfe505a16b3d86621444d11e90f901fbc9"}
Nov 02 14:30:38.460058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.460029    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:38.463405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.463383    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:38.552373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.552328    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-openshift-ca\") pod \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") "
Nov 02 14:30:38.552584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.552420    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-xks-ca\") pod \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") "
Nov 02 14:30:38.552584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.552449    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-kube-api-access-62qht\" (UniqueName: \"kubernetes.io/projected/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-kube-api-access-62qht\") pod \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") "
Nov 02 14:30:38.552689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.552619    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b6bd36-75d9-4a4e-a569-f1d181b84c13-kube-api-access-kcwmk\" (UniqueName: \"kubernetes.io/projected/71b6bd36-75d9-4a4e-a569-f1d181b84c13-kube-api-access-kcwmk\") pod \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") "
Nov 02 14:30:38.552793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.552764    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-xks-ca\") pod \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\" (UID: \"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3\") "
Nov 02 14:30:38.552876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.552825    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-openshift-ca\") pod \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\" (UID: \"71b6bd36-75d9-4a4e-a569-f1d181b84c13\") "
Nov 02 14:30:38.554285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.554251    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-openshift-ca" pod "71b6bd36-75d9-4a4e-a569-f1d181b84c13" (UID: "71b6bd36-75d9-4a4e-a569-f1d181b84c13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:38.554285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.554267    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-openshift-ca" pod "46d6b85e-f4cf-4ec3-aab9-d43d532c42f3" (UID: "46d6b85e-f4cf-4ec3-aab9-d43d532c42f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:30:38.566822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.557327    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b6bd36-75d9-4a4e-a569-f1d181b84c13-kube-api-access-kcwmk" pod "71b6bd36-75d9-4a4e-a569-f1d181b84c13" (UID: "71b6bd36-75d9-4a4e-a569-f1d181b84c13"). InnerVolumeSpecName "kube-api-access-kcwmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:38.566822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.557596    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-xks-ca" pod "71b6bd36-75d9-4a4e-a569-f1d181b84c13" (UID: "71b6bd36-75d9-4a4e-a569-f1d181b84c13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:30:38.566822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.558462    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-xks-ca" pod "46d6b85e-f4cf-4ec3-aab9-d43d532c42f3" (UID: "46d6b85e-f4cf-4ec3-aab9-d43d532c42f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:30:38.566822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.558573    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-kube-api-access-62qht" pod "46d6b85e-f4cf-4ec3-aab9-d43d532c42f3" (UID: "46d6b85e-f4cf-4ec3-aab9-d43d532c42f3"). InnerVolumeSpecName "kube-api-access-62qht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:30:38.655438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.655403    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:38.655438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.655435    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:38.655438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.655445    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:38.655698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.655454    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71b6bd36-75d9-4a4e-a569-f1d181b84c13-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:38.655698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.655462    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62qht\" (UniqueName: \"kubernetes.io/projected/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3-kube-api-access-62qht\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:38.655698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:38.655470    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcwmk\" (UniqueName: \"kubernetes.io/projected/71b6bd36-75d9-4a4e-a569-f1d181b84c13-kube-api-access-kcwmk\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:30:39.256456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:39.254591    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx" event={"ID":"46d6b85e-f4cf-4ec3-aab9-d43d532c42f3","Type":"ContainerDied","Data":"f6bb6562bbc3d8589976027e8de708087eb277163cfcce42dc6b5f11ef54adb9"}
Nov 02 14:30:39.256456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:39.254658    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6bb6562bbc3d8589976027e8de708087eb277163cfcce42dc6b5f11ef54adb9"
Nov 02 14:30:39.256456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:39.254748    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"
Nov 02 14:30:39.267834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:39.265328    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g" event={"ID":"71b6bd36-75d9-4a4e-a569-f1d181b84c13","Type":"ContainerDied","Data":"1b4fee41a7813f3973ec3021c80a7cfd83afa57724814c3d92ab019b14daab2e"}
Nov 02 14:30:39.267834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:39.265420    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b4fee41a7813f3973ec3021c80a7cfd83afa57724814c3d92ab019b14daab2e"
Nov 02 14:30:39.267834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:39.265494    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"
Nov 02 14:30:48.417908 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:30:48.417827    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:30:52.262763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.262718    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"]
Nov 02 14:30:52.264062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.264036    5034 cpu_manager.go:414] "containerMap: removing container" podUID="46d6b85e-f4cf-4ec3-aab9-d43d532c42f3" containerName="revoke-keys"
Nov 02 14:30:52.264062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.264062    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d6b85e-f4cf-4ec3-aab9-d43d532c42f3" containerName="revoke-keys"
Nov 02 14:30:52.264252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.264086    5034 cpu_manager.go:414] "containerMap: removing container" podUID="71b6bd36-75d9-4a4e-a569-f1d181b84c13" containerName="revoke-keys"
Nov 02 14:30:52.264252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.264094    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b6bd36-75d9-4a4e-a569-f1d181b84c13" containerName="revoke-keys"
Nov 02 14:30:52.264354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.264279    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d6b85e-f4cf-4ec3-aab9-d43d532c42f3" containerName="revoke-keys"
Nov 02 14:30:52.264354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.264302    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b6bd36-75d9-4a4e-a569-f1d181b84c13" containerName="revoke-keys"
Nov 02 14:30:52.271148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.271119    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.295320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.295277    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"]
Nov 02 14:30:52.405558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.405509    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.405758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.405566    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.405758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.405657    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsnjz\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-kube-api-access-vsnjz\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.405868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.405792    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.405928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.405912    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.406054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.406033    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.406106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.406091    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.406176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.406158    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.406269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.406252    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.507433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.507389    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.507625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.507472    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.507625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.507497    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.507625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.507520    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsnjz\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-kube-api-access-vsnjz\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.507820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.507638    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.507875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.507849    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.507875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.507865    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.508040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.508008    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.508131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.508090    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.508131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.508124    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.508256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.508150    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.508256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.507859    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.508256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.508124    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.508847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.508284    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.510602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.510572    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.510859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.510838    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.533154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.533069    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.533337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.533167    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsnjz\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-kube-api-access-vsnjz\") pod \"maas-default-gateway-openshift-default-6886b96f9-k7bdr\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.605282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.605253    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:52.982404 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:30:52.982364    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76c5fc52_0dad_47ac_ab36_f9ad186eabf8.slice/crio-254fb95421836f78c39ed875dd2b768ba8848bba155863da62fc85e6c821e833 WatchSource:0}: Error finding container 254fb95421836f78c39ed875dd2b768ba8848bba155863da62fc85e6c821e833: Status 404 returned error can't find the container with id 254fb95421836f78c39ed875dd2b768ba8848bba155863da62fc85e6c821e833
Nov 02 14:30:52.984404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.984377    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"]
Nov 02 14:30:52.986306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.986259    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:30:52.986433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.986351    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:30:52.986433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:52.986392    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:30:53.533274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:53.533227    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" event={"ID":"76c5fc52-0dad-47ac-ab36-f9ad186eabf8","Type":"ContainerStarted","Data":"ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb"}
Nov 02 14:30:53.533678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:53.533303    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" event={"ID":"76c5fc52-0dad-47ac-ab36-f9ad186eabf8","Type":"ContainerStarted","Data":"254fb95421836f78c39ed875dd2b768ba8848bba155863da62fc85e6c821e833"}
Nov 02 14:30:53.586421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:53.586357    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podStartSLOduration=1.586339716 podStartE2EDuration="1.586339716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:30:53.582485972 +0000 UTC m=+2844.333790413" watchObservedRunningTime="2026-11-02 14:30:53.586339716 +0000 UTC m=+2844.337644154"
Nov 02 14:30:53.605534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:53.605478    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:30:54.609320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:54.607870    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7bdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 14:30:54.609320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:54.607943    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 14:30:55.609927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:55.609195    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7bdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 14:30:55.609927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:55.609272    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 14:30:56.606933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:56.606613    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7bdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 14:30:56.606933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:56.606676    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 14:30:57.606636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:57.606580    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7bdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 14:30:57.607530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:57.607478    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 14:30:58.606347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:58.606307    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7bdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 14:30:58.606556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:58.606363    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 14:30:59.605801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:59.605643    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7bdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 14:30:59.605801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:30:59.605735    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 14:31:00.606408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:00.606371    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7bdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused" start-of-body=
Nov 02 14:31:00.610806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:00.607041    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": dial tcp 10.132.1.72:15021: connect: connection refused"
Nov 02 14:31:02.675799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:02.670871    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-k7bdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:31:02.675799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:02.670938    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:31:02.831617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:02.831580    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:31:02.832299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:02.832261    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:31:02.834157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:02.834130    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:31:18.418168 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:31:18.418126    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:31:23.343901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:23.343863    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"]
Nov 02 14:31:23.344429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:23.344216    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy" containerID="cri-o://c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2" gracePeriod=30
Nov 02 14:31:28.309280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.309239    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"]
Nov 02 14:31:28.309828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.309445    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy" containerID="cri-o://776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f" gracePeriod=30
Nov 02 14:31:28.320564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.320525    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:31:28.320732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.320607    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:31:28.654023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.653991    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:31:28.806332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806294    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istiod-ca-cert\") pod \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") "
Nov 02 14:31:28.806531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806363    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-envoy\") pod \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") "
Nov 02 14:31:28.806531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806422    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-podinfo\") pod \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") "
Nov 02 14:31:28.806531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806465    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-kube-api-access-gdtdh\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-kube-api-access-gdtdh\") pod \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") "
Nov 02 14:31:28.806707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806590    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-token\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-token\") pod \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") "
Nov 02 14:31:28.806707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806692    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-data\") pod \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") "
Nov 02 14:31:28.806839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806722    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istiod-ca-cert" pod "d2b8e4bb-ded4-47cc-b577-8312a7a6b502" (UID: "d2b8e4bb-ded4-47cc-b577-8312a7a6b502"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:31:28.806839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806754    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-certs\") pod \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") "
Nov 02 14:31:28.806953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806914    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-credential-socket\") pod \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") "
Nov 02 14:31:28.807025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.806955    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-socket\") pod \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\" (UID: \"d2b8e4bb-ded4-47cc-b577-8312a7a6b502\") "
Nov 02 14:31:28.807088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.807035    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-data" pod "d2b8e4bb-ded4-47cc-b577-8312a7a6b502" (UID: "d2b8e4bb-ded4-47cc-b577-8312a7a6b502"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:28.807146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.807099    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-certs" pod "d2b8e4bb-ded4-47cc-b577-8312a7a6b502" (UID: "d2b8e4bb-ded4-47cc-b577-8312a7a6b502"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:28.807359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.807257    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-credential-socket" pod "d2b8e4bb-ded4-47cc-b577-8312a7a6b502" (UID: "d2b8e4bb-ded4-47cc-b577-8312a7a6b502"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:28.807448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.807382    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-socket" pod "d2b8e4bb-ded4-47cc-b577-8312a7a6b502" (UID: "d2b8e4bb-ded4-47cc-b577-8312a7a6b502"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:28.807861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.807840    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:28.807909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.807869    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:28.807909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.807887    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:28.807909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.807901    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:28.808012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.807916    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:28.808960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.808909    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-token" pod "d2b8e4bb-ded4-47cc-b577-8312a7a6b502" (UID: "d2b8e4bb-ded4-47cc-b577-8312a7a6b502"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:31:28.809066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.809019    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-kube-api-access-gdtdh" pod "d2b8e4bb-ded4-47cc-b577-8312a7a6b502" (UID: "d2b8e4bb-ded4-47cc-b577-8312a7a6b502"). InnerVolumeSpecName "kube-api-access-gdtdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:31:28.809205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.809186    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-podinfo" pod "d2b8e4bb-ded4-47cc-b577-8312a7a6b502" (UID: "d2b8e4bb-ded4-47cc-b577-8312a7a6b502"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:31:28.809205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.809192    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-envoy" pod "d2b8e4bb-ded4-47cc-b577-8312a7a6b502" (UID: "d2b8e4bb-ded4-47cc-b577-8312a7a6b502"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:28.909862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.909772    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:28.909862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.909816    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:28.909862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.909825    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:28.909862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:28.909837    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdtdh\" (UniqueName: \"kubernetes.io/projected/d2b8e4bb-ded4-47cc-b577-8312a7a6b502-kube-api-access-gdtdh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:29.184345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.184242    5034 generic.go:356] "Generic (PLEG): container finished" podID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerID="c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2" exitCode=0
Nov 02 14:31:29.184345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.184292    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"
Nov 02 14:31:29.184345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.184324    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" event={"ID":"d2b8e4bb-ded4-47cc-b577-8312a7a6b502","Type":"ContainerDied","Data":"c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2"}
Nov 02 14:31:29.184626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.184383    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq" event={"ID":"d2b8e4bb-ded4-47cc-b577-8312a7a6b502","Type":"ContainerDied","Data":"5676b844233fdd75c9ecae7392937ae67d8fae256aa9f8593ec585e7e21f4456"}
Nov 02 14:31:29.184626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.184411    5034 scope.go:122] "RemoveContainer" containerID="c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2"
Nov 02 14:31:29.197369 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.197347    5034 scope.go:122] "RemoveContainer" containerID="c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2"
Nov 02 14:31:29.197712 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:31:29.197680    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2\": container with ID starting with c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2 not found: ID does not exist" containerID="c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2"
Nov 02 14:31:29.197836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.197718    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2"} err="failed to get container status \"c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2\": rpc error: code = NotFound desc = could not find container \"c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2\": container with ID starting with c0fcf5ad0624be7062c545a3019e07451135ad0dcf3db0d1e66a491619cb6bf2 not found: ID does not exist"
Nov 02 14:31:29.247228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.247192    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"]
Nov 02 14:31:29.259864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.259834    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-lh5vq"]
Nov 02 14:31:29.398540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:29.398509    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" path="/var/lib/kubelet/pods/d2b8e4bb-ded4-47cc-b577-8312a7a6b502/volumes"
Nov 02 14:31:33.617458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.617428    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:31:33.689867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.689830    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-socket\") pod \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") "
Nov 02 14:31:33.690080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.689920    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-credential-socket\") pod \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") "
Nov 02 14:31:33.690080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690019    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-kube-api-access-kppmc\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-kube-api-access-kppmc\") pod \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") "
Nov 02 14:31:33.690198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690089    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istiod-ca-cert\") pod \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") "
Nov 02 14:31:33.690198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690128    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-certs\") pod \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") "
Nov 02 14:31:33.690198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690141    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-socket" pod "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" (UID: "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:33.690198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690158    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-podinfo\") pod \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") "
Nov 02 14:31:33.691056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690536    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-credential-socket" pod "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" (UID: "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:33.691056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690663    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-envoy\") pod \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") "
Nov 02 14:31:33.691056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690691    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-certs" pod "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" (UID: "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:33.691056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690751    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istiod-ca-cert" pod "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" (UID: "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:31:33.691056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690914    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-token\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-token\") pod \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") "
Nov 02 14:31:33.691056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.690970    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-data\") pod \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\" (UID: \"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1\") "
Nov 02 14:31:33.691396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.691330    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-data" pod "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" (UID: "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:33.691702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.691683    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:33.691801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.691705    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:33.691801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.691714    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:33.691801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.691724    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:33.691801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.691733    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:33.693562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.693437    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-token" pod "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" (UID: "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:31:33.693562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.693524    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-kube-api-access-kppmc" pod "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" (UID: "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1"). InnerVolumeSpecName "kube-api-access-kppmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:31:33.694400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.694368    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-podinfo" pod "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" (UID: "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:31:33.694605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.694586    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-envoy" pod "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" (UID: "f5b1da07-8c07-4b41-af4a-33f5e79a1ad1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:31:33.801454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.799861    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kppmc\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-kube-api-access-kppmc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:33.801454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.799906    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:33.801454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.799922    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:33.801454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:33.799935    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:34.235122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:34.235088    5034 generic.go:356] "Generic (PLEG): container finished" podID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerID="776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f" exitCode=0
Nov 02 14:31:34.235320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:34.235135    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"
Nov 02 14:31:34.235320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:34.235179    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" event={"ID":"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1","Type":"ContainerDied","Data":"776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f"}
Nov 02 14:31:34.235320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:34.235242    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd" event={"ID":"f5b1da07-8c07-4b41-af4a-33f5e79a1ad1","Type":"ContainerDied","Data":"e3b18142527ba1355c857e4ff4ed4fc114f6dfe9cf1aa3afaaea28fe159e259a"}
Nov 02 14:31:34.235320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:34.235275    5034 scope.go:122] "RemoveContainer" containerID="776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f"
Nov 02 14:31:34.249806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:34.249766    5034 scope.go:122] "RemoveContainer" containerID="776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f"
Nov 02 14:31:34.251135 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:31:34.251112    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f\": container with ID starting with 776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f not found: ID does not exist" containerID="776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f"
Nov 02 14:31:34.251256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:34.251141    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f"} err="failed to get container status \"776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f\": rpc error: code = NotFound desc = could not find container \"776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f\": container with ID starting with 776050fc17e96157d334706477940292e08f08a36be50b60824c3f403880d38f not found: ID does not exist"
Nov 02 14:31:34.296492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:34.296460    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"]
Nov 02 14:31:34.311071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:34.311042    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-4c2811-openshift-default-5b5b8b6755-dmwfd"]
Nov 02 14:31:35.397631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:35.397600    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" path="/var/lib/kubelet/pods/f5b1da07-8c07-4b41-af4a-33f5e79a1ad1/volumes"
Nov 02 14:31:42.033321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:42.033285    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"]
Nov 02 14:31:42.046380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:42.046353    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bec254d5-hmbcc"]
Nov 02 14:31:43.399563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:43.399525    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4cc27d-dbf7-4596-98f6-d2efc5ce6891" path="/var/lib/kubelet/pods/cf4cc27d-dbf7-4596-98f6-d2efc5ce6891/volumes"
Nov 02 14:31:48.416974 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:31:48.416926    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:31:50.046258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.046213    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"]
Nov 02 14:31:50.057381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.057348    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4abe2db4-nhtzz"]
Nov 02 14:31:50.627846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.627808    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"]
Nov 02 14:31:50.628622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.628606    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy"
Nov 02 14:31:50.628622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.628622    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy"
Nov 02 14:31:50.628769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.628644    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy"
Nov 02 14:31:50.628769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.628650    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy"
Nov 02 14:31:50.628922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.628804    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b8e4bb-ded4-47cc-b577-8312a7a6b502" containerName="istio-proxy"
Nov 02 14:31:50.628922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.628818    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b1da07-8c07-4b41-af4a-33f5e79a1ad1" containerName="istio-proxy"
Nov 02 14:31:50.634057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.634036    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.653443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.653416    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"]
Nov 02 14:31:50.660137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.660115    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:31:50.660382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.660191    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:31:50.660382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.660293    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:31:50.665319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.665296    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:31:50.700357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.700329    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90e6d965-fcnpp\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.700506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.700384    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90e6d965-fcnpp\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.700614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.700595    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjsb\" (UniqueName: \"kubernetes.io/projected/cddd3528-d314-410d-a4f5-c092f1c42fa5-kube-api-access-wpjsb\") pod \"maas-api-revoke-sub-90e6d965-fcnpp\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.802637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.802598    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90e6d965-fcnpp\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.802841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.802673    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90e6d965-fcnpp\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.802921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.802853    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjsb\" (UniqueName: \"kubernetes.io/projected/cddd3528-d314-410d-a4f5-c092f1c42fa5-kube-api-access-wpjsb\") pod \"maas-api-revoke-sub-90e6d965-fcnpp\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.803323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.803300    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90e6d965-fcnpp\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.805239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.805213    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90e6d965-fcnpp\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.829132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.829107    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjsb\" (UniqueName: \"kubernetes.io/projected/cddd3528-d314-410d-a4f5-c092f1c42fa5-kube-api-access-wpjsb\") pod \"maas-api-revoke-sub-90e6d965-fcnpp\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:50.951005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:50.950917    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:51.259955 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:31:51.259916    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddd3528_d314_410d_a4f5_c092f1c42fa5.slice/crio-6385bf03742eccd886b8c911658fcbf8aba8c6b02b0f8bf320769a7a9103650c WatchSource:0}: Error finding container 6385bf03742eccd886b8c911658fcbf8aba8c6b02b0f8bf320769a7a9103650c: Status 404 returned error can't find the container with id 6385bf03742eccd886b8c911658fcbf8aba8c6b02b0f8bf320769a7a9103650c
Nov 02 14:31:51.265123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:51.265097    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"]
Nov 02 14:31:51.401012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:51.400977    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="296e26a3-3f71-4f75-89b9-4592599d94a3" path="/var/lib/kubelet/pods/296e26a3-3f71-4f75-89b9-4592599d94a3/volumes"
Nov 02 14:31:51.543932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:51.543855    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp" event={"ID":"cddd3528-d314-410d-a4f5-c092f1c42fa5","Type":"ContainerStarted","Data":"6385bf03742eccd886b8c911658fcbf8aba8c6b02b0f8bf320769a7a9103650c"}
Nov 02 14:31:52.039514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:52.039480    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"]
Nov 02 14:31:52.052970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:52.052943    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2f7d-4w7wq"]
Nov 02 14:31:52.555289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:52.555259    5034 generic.go:356] "Generic (PLEG): container finished" podID="cddd3528-d314-410d-a4f5-c092f1c42fa5" containerID="18de6e94ff659f4873165d735802390e67931d0cf657036021142e622342a65a" exitCode=0
Nov 02 14:31:52.555888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:52.555342    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp" event={"ID":"cddd3528-d314-410d-a4f5-c092f1c42fa5","Type":"ContainerDied","Data":"18de6e94ff659f4873165d735802390e67931d0cf657036021142e622342a65a"}
Nov 02 14:31:53.397612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.397581    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c367b83-3095-48d8-9a05-a481fc5bc516" path="/var/lib/kubelet/pods/1c367b83-3095-48d8-9a05-a481fc5bc516/volumes"
Nov 02 14:31:53.762587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.762560    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:53.771632 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.771609    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-xks-ca\") pod \"cddd3528-d314-410d-a4f5-c092f1c42fa5\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") "
Nov 02 14:31:53.771847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.771832    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-openshift-ca\") pod \"cddd3528-d314-410d-a4f5-c092f1c42fa5\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") "
Nov 02 14:31:53.771916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.771867    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cddd3528-d314-410d-a4f5-c092f1c42fa5-kube-api-access-wpjsb\" (UniqueName: \"kubernetes.io/projected/cddd3528-d314-410d-a4f5-c092f1c42fa5-kube-api-access-wpjsb\") pod \"cddd3528-d314-410d-a4f5-c092f1c42fa5\" (UID: \"cddd3528-d314-410d-a4f5-c092f1c42fa5\") "
Nov 02 14:31:53.772176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.772154    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-openshift-ca" pod "cddd3528-d314-410d-a4f5-c092f1c42fa5" (UID: "cddd3528-d314-410d-a4f5-c092f1c42fa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:31:53.772510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.772488    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:53.773640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.773619    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-xks-ca" pod "cddd3528-d314-410d-a4f5-c092f1c42fa5" (UID: "cddd3528-d314-410d-a4f5-c092f1c42fa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:31:53.774062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.774034    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddd3528-d314-410d-a4f5-c092f1c42fa5-kube-api-access-wpjsb" pod "cddd3528-d314-410d-a4f5-c092f1c42fa5" (UID: "cddd3528-d314-410d-a4f5-c092f1c42fa5"). InnerVolumeSpecName "kube-api-access-wpjsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:31:53.874354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.874314    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddd3528-d314-410d-a4f5-c092f1c42fa5-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:53.874354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:53.874354    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpjsb\" (UniqueName: \"kubernetes.io/projected/cddd3528-d314-410d-a4f5-c092f1c42fa5-kube-api-access-wpjsb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:31:54.578071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:54.578041    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"
Nov 02 14:31:54.578071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:54.578051    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp" event={"ID":"cddd3528-d314-410d-a4f5-c092f1c42fa5","Type":"ContainerDied","Data":"6385bf03742eccd886b8c911658fcbf8aba8c6b02b0f8bf320769a7a9103650c"}
Nov 02 14:31:54.578337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:31:54.578124    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6385bf03742eccd886b8c911658fcbf8aba8c6b02b0f8bf320769a7a9103650c"
Nov 02 14:32:01.039062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.039013    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"]
Nov 02 14:32:01.052360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.052331    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0183dc5-ctl6n"]
Nov 02 14:32:01.345601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.345515    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"]
Nov 02 14:32:01.346493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.346472    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cddd3528-d314-410d-a4f5-c092f1c42fa5" containerName="revoke-keys"
Nov 02 14:32:01.346493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.346495    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cddd3528-d314-410d-a4f5-c092f1c42fa5" containerName="revoke-keys"
Nov 02 14:32:01.346673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.346662    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cddd3528-d314-410d-a4f5-c092f1c42fa5" containerName="revoke-keys"
Nov 02 14:32:01.350757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.350739    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.367667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.367640    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:32:01.367667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.367641    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:32:01.367908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.367891    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:32:01.367968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.367892    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:32:01.369122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.369101    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"]
Nov 02 14:32:01.406263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.406226    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgs7t\" (UniqueName: \"kubernetes.io/projected/c55b7f19-c693-4fa2-9c01-03230f0f5924-kube-api-access-tgs7t\") pod \"maas-api-revoke-sub-a6a2f60a-vfg9d\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.406564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.406542    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a2f60a-vfg9d\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.406691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.406605    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a2f60a-vfg9d\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.406769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.406709    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10933ec6-c93c-42bd-aaf2-820dfd803ce1" path="/var/lib/kubelet/pods/10933ec6-c93c-42bd-aaf2-820dfd803ce1/volumes"
Nov 02 14:32:01.507966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.507928    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgs7t\" (UniqueName: \"kubernetes.io/projected/c55b7f19-c693-4fa2-9c01-03230f0f5924-kube-api-access-tgs7t\") pod \"maas-api-revoke-sub-a6a2f60a-vfg9d\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.508161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.508032    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a2f60a-vfg9d\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.508161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.508074    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a2f60a-vfg9d\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.508751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.508725    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a2f60a-vfg9d\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.510595 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.510570    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a2f60a-vfg9d\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.533216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.533187    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgs7t\" (UniqueName: \"kubernetes.io/projected/c55b7f19-c693-4fa2-9c01-03230f0f5924-kube-api-access-tgs7t\") pod \"maas-api-revoke-sub-a6a2f60a-vfg9d\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.668825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.668691    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:01.982325 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:32:01.982290    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55b7f19_c693_4fa2_9c01_03230f0f5924.slice/crio-48364827cd37d1b22a7174651468c30f9bc367d4e125283caae01bb44fb5acf9 WatchSource:0}: Error finding container 48364827cd37d1b22a7174651468c30f9bc367d4e125283caae01bb44fb5acf9: Status 404 returned error can't find the container with id 48364827cd37d1b22a7174651468c30f9bc367d4e125283caae01bb44fb5acf9
Nov 02 14:32:01.985585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:01.985472    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"]
Nov 02 14:32:02.693865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:02.687969    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d" event={"ID":"c55b7f19-c693-4fa2-9c01-03230f0f5924","Type":"ContainerStarted","Data":"ab95953f506f017cbbd34f649e6e332c71b13a093e81bc0268729747a0768aaf"}
Nov 02 14:32:02.693865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:02.688059    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d" event={"ID":"c55b7f19-c693-4fa2-9c01-03230f0f5924","Type":"ContainerStarted","Data":"48364827cd37d1b22a7174651468c30f9bc367d4e125283caae01bb44fb5acf9"}
Nov 02 14:32:02.737425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:02.737369    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d" podStartSLOduration=1.7373546800000002 podStartE2EDuration="1.73735468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:02.734237241 +0000 UTC m=+2913.485541680" watchObservedRunningTime="2026-11-02 14:32:02.73735468 +0000 UTC m=+2913.488659117"
Nov 02 14:32:03.700539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:03.700501    5034 generic.go:356] "Generic (PLEG): container finished" podID="c55b7f19-c693-4fa2-9c01-03230f0f5924" containerID="ab95953f506f017cbbd34f649e6e332c71b13a093e81bc0268729747a0768aaf" exitCode=0
Nov 02 14:32:03.700969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:03.700584    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d" event={"ID":"c55b7f19-c693-4fa2-9c01-03230f0f5924","Type":"ContainerDied","Data":"ab95953f506f017cbbd34f649e6e332c71b13a093e81bc0268729747a0768aaf"}
Nov 02 14:32:04.913492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:04.913464    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:05.036466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.036430    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"]
Nov 02 14:32:05.044890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.044854    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3888b8c-mh647"]
Nov 02 14:32:05.074164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.074132    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-openshift-ca\") pod \"c55b7f19-c693-4fa2-9c01-03230f0f5924\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") "
Nov 02 14:32:05.074386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.074366    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-xks-ca\") pod \"c55b7f19-c693-4fa2-9c01-03230f0f5924\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") "
Nov 02 14:32:05.074542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.074526    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55b7f19-c693-4fa2-9c01-03230f0f5924-kube-api-access-tgs7t\" (UniqueName: \"kubernetes.io/projected/c55b7f19-c693-4fa2-9c01-03230f0f5924-kube-api-access-tgs7t\") pod \"c55b7f19-c693-4fa2-9c01-03230f0f5924\" (UID: \"c55b7f19-c693-4fa2-9c01-03230f0f5924\") "
Nov 02 14:32:05.074689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.074553    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-openshift-ca" pod "c55b7f19-c693-4fa2-9c01-03230f0f5924" (UID: "c55b7f19-c693-4fa2-9c01-03230f0f5924"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:32:05.075302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.075279    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:05.077096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.077068    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-xks-ca" pod "c55b7f19-c693-4fa2-9c01-03230f0f5924" (UID: "c55b7f19-c693-4fa2-9c01-03230f0f5924"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:32:05.077244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.077225    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55b7f19-c693-4fa2-9c01-03230f0f5924-kube-api-access-tgs7t" pod "c55b7f19-c693-4fa2-9c01-03230f0f5924" (UID: "c55b7f19-c693-4fa2-9c01-03230f0f5924"). InnerVolumeSpecName "kube-api-access-tgs7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:32:05.177105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.177065    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgs7t\" (UniqueName: \"kubernetes.io/projected/c55b7f19-c693-4fa2-9c01-03230f0f5924-kube-api-access-tgs7t\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:05.177105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.177101    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c55b7f19-c693-4fa2-9c01-03230f0f5924-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:05.400652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.400566    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b799b069-64ec-49e5-a028-0dc5388f88cf" path="/var/lib/kubelet/pods/b799b069-64ec-49e5-a028-0dc5388f88cf/volumes"
Nov 02 14:32:05.724831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.724716    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"
Nov 02 14:32:05.724831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.724756    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d" event={"ID":"c55b7f19-c693-4fa2-9c01-03230f0f5924","Type":"ContainerDied","Data":"48364827cd37d1b22a7174651468c30f9bc367d4e125283caae01bb44fb5acf9"}
Nov 02 14:32:05.725034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:05.724844    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48364827cd37d1b22a7174651468c30f9bc367d4e125283caae01bb44fb5acf9"
Nov 02 14:32:18.421609 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:32:18.421524    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:32:36.397937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.397904    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"]
Nov 02 14:32:36.400471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.398746    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c55b7f19-c693-4fa2-9c01-03230f0f5924" containerName="revoke-keys"
Nov 02 14:32:36.400471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.398771    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55b7f19-c693-4fa2-9c01-03230f0f5924" containerName="revoke-keys"
Nov 02 14:32:36.400471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.399021    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55b7f19-c693-4fa2-9c01-03230f0f5924" containerName="revoke-keys"
Nov 02 14:32:36.403412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.403395    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.426053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.426010    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"]
Nov 02 14:32:36.426581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.426554    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:32:36.426871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.426846    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:32:36.426980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.426900    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:32:36.430309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.430284    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:32:36.503283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.503246    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vmd4\" (UniqueName: \"kubernetes.io/projected/b615f723-80d6-4377-a4e4-70398bc891ac-kube-api-access-4vmd4\") pod \"maas-api-revoke-sub-78649acf-s425c\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.503627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.503597    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78649acf-s425c\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.503879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.503854    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78649acf-s425c\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.605764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.605725    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vmd4\" (UniqueName: \"kubernetes.io/projected/b615f723-80d6-4377-a4e4-70398bc891ac-kube-api-access-4vmd4\") pod \"maas-api-revoke-sub-78649acf-s425c\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.605990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.605829    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78649acf-s425c\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.605990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.605962    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78649acf-s425c\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.606513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.606489    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78649acf-s425c\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.608395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.608373    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78649acf-s425c\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.638726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.638702    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vmd4\" (UniqueName: \"kubernetes.io/projected/b615f723-80d6-4377-a4e4-70398bc891ac-kube-api-access-4vmd4\") pod \"maas-api-revoke-sub-78649acf-s425c\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:36.720892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:36.720827    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:37.105277 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:32:37.105228    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb615f723_80d6_4377_a4e4_70398bc891ac.slice/crio-5b49e5a2f4db42f9df07820bb9ef445d6b3d28d817d1ace67a371a434c027ed6 WatchSource:0}: Error finding container 5b49e5a2f4db42f9df07820bb9ef445d6b3d28d817d1ace67a371a434c027ed6: Status 404 returned error can't find the container with id 5b49e5a2f4db42f9df07820bb9ef445d6b3d28d817d1ace67a371a434c027ed6
Nov 02 14:32:37.117298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:37.117269    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"]
Nov 02 14:32:37.731799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:37.731744    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c" event={"ID":"b615f723-80d6-4377-a4e4-70398bc891ac","Type":"ContainerStarted","Data":"53f809c2cb0850f1531e1e1e0e1609740b512c8bd7bd37a72f8d0ec492600c66"}
Nov 02 14:32:37.733353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:37.733326    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c" event={"ID":"b615f723-80d6-4377-a4e4-70398bc891ac","Type":"ContainerStarted","Data":"5b49e5a2f4db42f9df07820bb9ef445d6b3d28d817d1ace67a371a434c027ed6"}
Nov 02 14:32:37.791534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:37.791463    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c" podStartSLOduration=1.791443883 podStartE2EDuration="1.791443883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:37.784030124 +0000 UTC m=+2948.535334546" watchObservedRunningTime="2026-11-02 14:32:37.791443883 +0000 UTC m=+2948.542748322"
Nov 02 14:32:38.767063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:38.767015    5034 generic.go:356] "Generic (PLEG): container finished" podID="b615f723-80d6-4377-a4e4-70398bc891ac" containerID="53f809c2cb0850f1531e1e1e0e1609740b512c8bd7bd37a72f8d0ec492600c66" exitCode=0
Nov 02 14:32:38.768270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:38.768234    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c" event={"ID":"b615f723-80d6-4377-a4e4-70398bc891ac","Type":"ContainerDied","Data":"53f809c2cb0850f1531e1e1e0e1609740b512c8bd7bd37a72f8d0ec492600c66"}
Nov 02 14:32:39.965167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:39.965140    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:40.018873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.016340    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b615f723-80d6-4377-a4e4-70398bc891ac-kube-api-access-4vmd4\" (UniqueName: \"kubernetes.io/projected/b615f723-80d6-4377-a4e4-70398bc891ac-kube-api-access-4vmd4\") pod \"b615f723-80d6-4377-a4e4-70398bc891ac\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") "
Nov 02 14:32:40.018873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.016487    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-xks-ca\") pod \"b615f723-80d6-4377-a4e4-70398bc891ac\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") "
Nov 02 14:32:40.018873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.016628    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-openshift-ca\") pod \"b615f723-80d6-4377-a4e4-70398bc891ac\" (UID: \"b615f723-80d6-4377-a4e4-70398bc891ac\") "
Nov 02 14:32:40.018873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.017934    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-openshift-ca" pod "b615f723-80d6-4377-a4e4-70398bc891ac" (UID: "b615f723-80d6-4377-a4e4-70398bc891ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:32:40.021192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.021151    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-xks-ca" pod "b615f723-80d6-4377-a4e4-70398bc891ac" (UID: "b615f723-80d6-4377-a4e4-70398bc891ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:32:40.021345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.021327    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b615f723-80d6-4377-a4e4-70398bc891ac-kube-api-access-4vmd4" pod "b615f723-80d6-4377-a4e4-70398bc891ac" (UID: "b615f723-80d6-4377-a4e4-70398bc891ac"). InnerVolumeSpecName "kube-api-access-4vmd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:32:40.119138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.119093    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:40.119138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.119134    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b615f723-80d6-4377-a4e4-70398bc891ac-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:40.119384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.119150    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vmd4\" (UniqueName: \"kubernetes.io/projected/b615f723-80d6-4377-a4e4-70398bc891ac-kube-api-access-4vmd4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:40.789469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.789437    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"
Nov 02 14:32:40.789668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.789477    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c" event={"ID":"b615f723-80d6-4377-a4e4-70398bc891ac","Type":"ContainerDied","Data":"5b49e5a2f4db42f9df07820bb9ef445d6b3d28d817d1ace67a371a434c027ed6"}
Nov 02 14:32:40.789668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:40.789532    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b49e5a2f4db42f9df07820bb9ef445d6b3d28d817d1ace67a371a434c027ed6"
Nov 02 14:32:47.039989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:47.039954    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"]
Nov 02 14:32:47.052936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:47.052909    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e80b02f5-nq6g8"]
Nov 02 14:32:47.397242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:47.397167    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a4de6d4-6060-4cee-b97f-6537e38a7592" path="/var/lib/kubelet/pods/1a4de6d4-6060-4cee-b97f-6537e38a7592/volumes"
Nov 02 14:32:48.060988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.060956    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"]
Nov 02 14:32:48.061815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.061799    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b615f723-80d6-4377-a4e4-70398bc891ac" containerName="revoke-keys"
Nov 02 14:32:48.061815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.061815    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b615f723-80d6-4377-a4e4-70398bc891ac" containerName="revoke-keys"
Nov 02 14:32:48.061978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.061969    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b615f723-80d6-4377-a4e4-70398bc891ac" containerName="revoke-keys"
Nov 02 14:32:48.066342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.066319    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.089115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.089089    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"]
Nov 02 14:32:48.094700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.094679    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f8ae8e27fe78\""
Nov 02 14:32:48.094876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.094684    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 14:32:48.094876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.094688    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 14:32:48.097828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.097807    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 14:32:48.097967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.097816    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 14:32:48.098051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.097824    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-glvff\""
Nov 02 14:32:48.169357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.169326    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-plugins-config-volume\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.169569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.169553    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-routing-overlay-volume\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.169735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.169718    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzq4c\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-kube-api-access-vzq4c\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.169797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.169751    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-provider-credentials\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.270669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.270639    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-plugins-config-volume\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.270844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.270763    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-routing-overlay-volume\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.270915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.270901    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzq4c\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-kube-api-access-vzq4c\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.270963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.270923    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-provider-credentials\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.271418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.271400    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-plugins-config-volume\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.271483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.271438    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-routing-overlay-volume\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.273393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.273376    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-provider-credentials\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.295852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.295831    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzq4c\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-kube-api-access-vzq4c\") pod \"payload-processing-external-model-5698cdb5c5-ndhbf\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.389270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.389174    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:48.494400 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:32:48.494357    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:32:48.790117 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:32:48.790082    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59b9037_0553_43ac_80af_864b88c057d0.slice/crio-0b2d30c72f7c73d308c095091cd3fbe883e182d9a75bdaf13f4b8bdd7c0b892b WatchSource:0}: Error finding container 0b2d30c72f7c73d308c095091cd3fbe883e182d9a75bdaf13f4b8bdd7c0b892b: Status 404 returned error can't find the container with id 0b2d30c72f7c73d308c095091cd3fbe883e182d9a75bdaf13f4b8bdd7c0b892b
Nov 02 14:32:48.793314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.793124    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"]
Nov 02 14:32:48.869608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:48.869576    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf" event={"ID":"e59b9037-0553-43ac-80af-864b88c057d0","Type":"ContainerStarted","Data":"0b2d30c72f7c73d308c095091cd3fbe883e182d9a75bdaf13f4b8bdd7c0b892b"}
Nov 02 14:32:49.968816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:49.964952    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf" event={"ID":"e59b9037-0553-43ac-80af-864b88c057d0","Type":"ContainerStarted","Data":"00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74"}
Nov 02 14:32:49.968816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:49.966221    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:50.336802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:50.335822    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf" podStartSLOduration=2.335796795 podStartE2EDuration="2.335796795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:50.029200628 +0000 UTC m=+2960.780505079" watchObservedRunningTime="2026-11-02 14:32:50.335796795 +0000 UTC m=+2961.087101238"
Nov 02 14:32:51.293927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.293894    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"]
Nov 02 14:32:51.320364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.320325    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"]
Nov 02 14:32:51.326285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.326253    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.345641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.345570    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-75ba6fa2a15d\""
Nov 02 14:32:51.348448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.348427    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"]
Nov 02 14:32:51.380258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.380227    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nb5l\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-kube-api-access-5nb5l\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.380421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.380273    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-provider-credentials\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.380421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.380298    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-plugins-config-volume\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.380591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.380567    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-routing-overlay-volume\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.482086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.482054    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-plugins-config-volume\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.482246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.482191    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-routing-overlay-volume\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.482488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.482467    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nb5l\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-kube-api-access-5nb5l\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.482562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.482520    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-provider-credentials\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.482952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.482927    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-routing-overlay-volume\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.483070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.482930    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-plugins-config-volume\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.485281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.485256    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-provider-credentials\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.510088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.510065    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nb5l\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-kube-api-access-5nb5l\") pod \"payload-processing-external-model-6db49ff4fd-h7lj8\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.647670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.647593    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:51.965141 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:32:51.965111    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57543a3_d2fe_4d00_a7bc_740482a1593a.slice/crio-2e168d633f4b4df5ff44002d6baded3302e416ff2a539511e5be61c3f00ba03a WatchSource:0}: Error finding container 2e168d633f4b4df5ff44002d6baded3302e416ff2a539511e5be61c3f00ba03a: Status 404 returned error can't find the container with id 2e168d633f4b4df5ff44002d6baded3302e416ff2a539511e5be61c3f00ba03a
Nov 02 14:32:51.967756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.967725    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"]
Nov 02 14:32:51.985101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:51.985072    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8" event={"ID":"d57543a3-d2fe-4d00-a7bc-740482a1593a","Type":"ContainerStarted","Data":"2e168d633f4b4df5ff44002d6baded3302e416ff2a539511e5be61c3f00ba03a"}
Nov 02 14:32:52.996190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:52.996154    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8" event={"ID":"d57543a3-d2fe-4d00-a7bc-740482a1593a","Type":"ContainerStarted","Data":"611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb"}
Nov 02 14:32:52.996771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:52.996266    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:32:53.045043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:53.044995    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8" podStartSLOduration=2.044982877 podStartE2EDuration="2.044982877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:53.040925831 +0000 UTC m=+2963.792230271" watchObservedRunningTime="2026-11-02 14:32:53.044982877 +0000 UTC m=+2963.796287315"
Nov 02 14:32:56.267870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.267835    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"]
Nov 02 14:32:56.299855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.299820    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"]
Nov 02 14:32:56.305365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.305346    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.325080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.325051    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"]
Nov 02 14:32:56.390802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.386163    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nbhw\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-kube-api-access-9nbhw\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.390802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.386650    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-plugins-config-volume\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.390802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.386753    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-routing-overlay-volume\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.390802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.386798    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-provider-credentials\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.491233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.491191    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-plugins-config-volume\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.492522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.492454    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-routing-overlay-volume\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.493676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.493064    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-provider-credentials\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.493676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.493111    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nbhw\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-kube-api-access-9nbhw\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.494411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.494374    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-routing-overlay-volume\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.494600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.492815    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-plugins-config-volume\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.497106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.497086    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-provider-credentials\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.570851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.565558    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nbhw\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-kube-api-access-9nbhw\") pod \"payload-processing-external-model-5698cdb5c5-slhbd\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:56.626802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:56.626138    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:57.011505 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:32:57.011470    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4143f67_1e77_448f_a1a6_4e0887be109b.slice/crio-b5483fe84ff6578251b54cc96a7ab2c391526bccf4f68bdeb7d635090c5722ee WatchSource:0}: Error finding container b5483fe84ff6578251b54cc96a7ab2c391526bccf4f68bdeb7d635090c5722ee: Status 404 returned error can't find the container with id b5483fe84ff6578251b54cc96a7ab2c391526bccf4f68bdeb7d635090c5722ee
Nov 02 14:32:57.014433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:57.014407    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"]
Nov 02 14:32:57.040719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:57.040684    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd" event={"ID":"a4143f67-1e77-448f-a1a6-4e0887be109b","Type":"ContainerStarted","Data":"b5483fe84ff6578251b54cc96a7ab2c391526bccf4f68bdeb7d635090c5722ee"}
Nov 02 14:32:57.997000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:57.996957    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf" podUID="e59b9037-0553-43ac-80af-864b88c057d0" containerName="payload-processing" containerID="cri-o://00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74" gracePeriod=25
Nov 02 14:32:58.054078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.054049    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd" event={"ID":"a4143f67-1e77-448f-a1a6-4e0887be109b","Type":"ContainerStarted","Data":"45418787199ddba63eaf8c0851a1555c8a002e4c5c6220d0c8a0bba094a4f626"}
Nov 02 14:32:58.054199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.054133    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:32:58.100287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.100225    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd" podStartSLOduration=2.100205667 podStartE2EDuration="2.100205667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:32:58.098590608 +0000 UTC m=+2968.849895047" watchObservedRunningTime="2026-11-02 14:32:58.100205667 +0000 UTC m=+2968.851510106"
Nov 02 14:32:58.303115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.303087    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:58.441479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.441443    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-provider-credentials\") pod \"e59b9037-0553-43ac-80af-864b88c057d0\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") "
Nov 02 14:32:58.441676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.441546    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-plugins-config-volume\") pod \"e59b9037-0553-43ac-80af-864b88c057d0\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") "
Nov 02 14:32:58.441763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.441697    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-routing-overlay-volume\") pod \"e59b9037-0553-43ac-80af-864b88c057d0\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") "
Nov 02 14:32:58.441763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.441732    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-kube-api-access-vzq4c\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-kube-api-access-vzq4c\") pod \"e59b9037-0553-43ac-80af-864b88c057d0\" (UID: \"e59b9037-0553-43ac-80af-864b88c057d0\") "
Nov 02 14:32:58.442051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.442019    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-plugins-config-volume" pod "e59b9037-0553-43ac-80af-864b88c057d0" (UID: "e59b9037-0553-43ac-80af-864b88c057d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:32:58.442132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.442108    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-routing-overlay-volume" pod "e59b9037-0553-43ac-80af-864b88c057d0" (UID: "e59b9037-0553-43ac-80af-864b88c057d0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:32:58.442517 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.442497    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:58.442517 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.442516    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e59b9037-0553-43ac-80af-864b88c057d0-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:58.443588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.443566    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-provider-credentials" pod "e59b9037-0553-43ac-80af-864b88c057d0" (UID: "e59b9037-0553-43ac-80af-864b88c057d0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:32:58.443953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.443935    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-kube-api-access-vzq4c" pod "e59b9037-0553-43ac-80af-864b88c057d0" (UID: "e59b9037-0553-43ac-80af-864b88c057d0"). InnerVolumeSpecName "kube-api-access-vzq4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:32:58.544303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.544197    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzq4c\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-kube-api-access-vzq4c\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:58.544303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:58.544227    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e59b9037-0553-43ac-80af-864b88c057d0-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:32:59.064463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.064424    5034 generic.go:356] "Generic (PLEG): container finished" podID="e59b9037-0553-43ac-80af-864b88c057d0" containerID="00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74" exitCode=0
Nov 02 14:32:59.064984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.064468    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"
Nov 02 14:32:59.064984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.064503    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf" event={"ID":"e59b9037-0553-43ac-80af-864b88c057d0","Type":"ContainerDied","Data":"00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74"}
Nov 02 14:32:59.064984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.064562    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf" event={"ID":"e59b9037-0553-43ac-80af-864b88c057d0","Type":"ContainerDied","Data":"0b2d30c72f7c73d308c095091cd3fbe883e182d9a75bdaf13f4b8bdd7c0b892b"}
Nov 02 14:32:59.064984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.064585    5034 scope.go:122] "RemoveContainer" containerID="00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74"
Nov 02 14:32:59.079376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.079352    5034 scope.go:122] "RemoveContainer" containerID="00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74"
Nov 02 14:32:59.079713 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:32:59.079681    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74\": container with ID starting with 00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74 not found: ID does not exist" containerID="00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74"
Nov 02 14:32:59.079851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.079723    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74"} err="failed to get container status \"00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74\": rpc error: code = NotFound desc = could not find container \"00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74\": container with ID starting with 00fbc730ff06eee872d7ec266801be35460b00f9d0c7096b2e853ae1146cac74 not found: ID does not exist"
Nov 02 14:32:59.126991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.126953    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"]
Nov 02 14:32:59.145116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.144672    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-ndhbf"]
Nov 02 14:32:59.402952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:32:59.402874    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59b9037-0553-43ac-80af-864b88c057d0" path="/var/lib/kubelet/pods/e59b9037-0553-43ac-80af-864b88c057d0/volumes"
Nov 02 14:33:00.090398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:00.090333    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"]
Nov 02 14:33:00.104981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:00.104947    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcb5276-cr28w"]
Nov 02 14:33:01.269207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.269167    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8" podUID="d57543a3-d2fe-4d00-a7bc-740482a1593a" containerName="payload-processing" containerID="cri-o://611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb" gracePeriod=25
Nov 02 14:33:01.400702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.400664    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6a675a-9567-48bf-aa35-07099224550a" path="/var/lib/kubelet/pods/6b6a675a-9567-48bf-aa35-07099224550a/volumes"
Nov 02 14:33:01.637949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.637920    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:33:01.806096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.806060    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-plugins-config-volume\") pod \"d57543a3-d2fe-4d00-a7bc-740482a1593a\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") "
Nov 02 14:33:01.806274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.806257    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-routing-overlay-volume\") pod \"d57543a3-d2fe-4d00-a7bc-740482a1593a\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") "
Nov 02 14:33:01.806469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.806454    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-kube-api-access-5nb5l\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-kube-api-access-5nb5l\") pod \"d57543a3-d2fe-4d00-a7bc-740482a1593a\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") "
Nov 02 14:33:01.806530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.806514    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-provider-credentials\") pod \"d57543a3-d2fe-4d00-a7bc-740482a1593a\" (UID: \"d57543a3-d2fe-4d00-a7bc-740482a1593a\") "
Nov 02 14:33:01.806612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.806569    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-plugins-config-volume" pod "d57543a3-d2fe-4d00-a7bc-740482a1593a" (UID: "d57543a3-d2fe-4d00-a7bc-740482a1593a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:01.806679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.806638    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-routing-overlay-volume" pod "d57543a3-d2fe-4d00-a7bc-740482a1593a" (UID: "d57543a3-d2fe-4d00-a7bc-740482a1593a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:01.807297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.807272    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:01.807297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.807295    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d57543a3-d2fe-4d00-a7bc-740482a1593a-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:01.808731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.808702    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-provider-credentials" pod "d57543a3-d2fe-4d00-a7bc-740482a1593a" (UID: "d57543a3-d2fe-4d00-a7bc-740482a1593a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:01.808838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.808726    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-kube-api-access-5nb5l" pod "d57543a3-d2fe-4d00-a7bc-740482a1593a" (UID: "d57543a3-d2fe-4d00-a7bc-740482a1593a"). InnerVolumeSpecName "kube-api-access-5nb5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:01.909224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.909124    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nb5l\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-kube-api-access-5nb5l\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:01.909224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:01.909164    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d57543a3-d2fe-4d00-a7bc-740482a1593a-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:02.116609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:02.116572    5034 generic.go:356] "Generic (PLEG): container finished" podID="d57543a3-d2fe-4d00-a7bc-740482a1593a" containerID="611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb" exitCode=0
Nov 02 14:33:02.116842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:02.116642    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"
Nov 02 14:33:02.116934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:02.116861    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8" event={"ID":"d57543a3-d2fe-4d00-a7bc-740482a1593a","Type":"ContainerDied","Data":"611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb"}
Nov 02 14:33:02.116934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:02.116915    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8" event={"ID":"d57543a3-d2fe-4d00-a7bc-740482a1593a","Type":"ContainerDied","Data":"2e168d633f4b4df5ff44002d6baded3302e416ff2a539511e5be61c3f00ba03a"}
Nov 02 14:33:02.117658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:02.116947    5034 scope.go:122] "RemoveContainer" containerID="611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb"
Nov 02 14:33:02.130287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:02.130264    5034 scope.go:122] "RemoveContainer" containerID="611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb"
Nov 02 14:33:02.130604 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:33:02.130581    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb\": container with ID starting with 611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb not found: ID does not exist" containerID="611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb"
Nov 02 14:33:02.130651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:02.130616    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb"} err="failed to get container status \"611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb\": rpc error: code = NotFound desc = could not find container \"611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb\": container with ID starting with 611baa86cd82806e192c1b3f92171c6cf54c3b9317d9b25c83a1178b641ae6eb not found: ID does not exist"
Nov 02 14:33:02.178027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:02.177937    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"]
Nov 02 14:33:02.191310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:02.191272    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-h7lj8"]
Nov 02 14:33:03.399473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.399433    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57543a3-d2fe-4d00-a7bc-740482a1593a" path="/var/lib/kubelet/pods/d57543a3-d2fe-4d00-a7bc-740482a1593a/volumes"
Nov 02 14:33:03.593010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.592971    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"]
Nov 02 14:33:03.596264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.596235    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:33:03.627544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.627510    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"]
Nov 02 14:33:03.628379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.628363    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d57543a3-d2fe-4d00-a7bc-740482a1593a" containerName="payload-processing"
Nov 02 14:33:03.628438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.628380    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57543a3-d2fe-4d00-a7bc-740482a1593a" containerName="payload-processing"
Nov 02 14:33:03.628438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.628429    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e59b9037-0553-43ac-80af-864b88c057d0" containerName="payload-processing"
Nov 02 14:33:03.628438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.628436    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59b9037-0553-43ac-80af-864b88c057d0" containerName="payload-processing"
Nov 02 14:33:03.628567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.628558    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57543a3-d2fe-4d00-a7bc-740482a1593a" containerName="payload-processing"
Nov 02 14:33:03.628605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.628578    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59b9037-0553-43ac-80af-864b88c057d0" containerName="payload-processing"
Nov 02 14:33:03.634268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.634247    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.654491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.654427    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-75ba6fa2a15d\""
Nov 02 14:33:03.660857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.660821    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"]
Nov 02 14:33:03.774419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.774373    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-routing-overlay-volume\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.774611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.774534    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-provider-credentials\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.774730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.774714    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lnv9\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-kube-api-access-9lnv9\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.774811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.774796    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-plugins-config-volume\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.876605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.876567    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-routing-overlay-volume\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.876840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.876654    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-provider-credentials\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.876840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.876812    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lnv9\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-kube-api-access-9lnv9\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.876981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.876870    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-plugins-config-volume\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.877419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.877391    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-routing-overlay-volume\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.877677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.877658    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-plugins-config-volume\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.879352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.879330    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-provider-credentials\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.902253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.902225    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lnv9\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-kube-api-access-9lnv9\") pod \"payload-processing-external-model-6db49ff4fd-jm7jt\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") " pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:03.950989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:03.950900    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:04.274465 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:33:04.274428    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8175053c_ddb7_46dd_9970_4f3e996d3e64.slice/crio-b39c814e76f453beb143313a6316c315ab35986a9b712d312997bd2d5d350759 WatchSource:0}: Error finding container b39c814e76f453beb143313a6316c315ab35986a9b712d312997bd2d5d350759: Status 404 returned error can't find the container with id b39c814e76f453beb143313a6316c315ab35986a9b712d312997bd2d5d350759
Nov 02 14:33:04.279943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:04.279918    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"]
Nov 02 14:33:05.150913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:05.150881    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt" event={"ID":"8175053c-ddb7-46dd-9970-4f3e996d3e64","Type":"ContainerStarted","Data":"87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca"}
Nov 02 14:33:05.151379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:05.150933    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt" event={"ID":"8175053c-ddb7-46dd-9970-4f3e996d3e64","Type":"ContainerStarted","Data":"b39c814e76f453beb143313a6316c315ab35986a9b712d312997bd2d5d350759"}
Nov 02 14:33:05.151379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:05.151123    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:05.253817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:05.250272    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt" podStartSLOduration=2.250249589 podStartE2EDuration="2.250249589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:05.216124249 +0000 UTC m=+2975.967428688" watchObservedRunningTime="2026-11-02 14:33:05.250249589 +0000 UTC m=+2976.001554028"
Nov 02 14:33:08.597800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:08.593855    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd" podUID="a4143f67-1e77-448f-a1a6-4e0887be109b" containerName="payload-processing" containerID="cri-o://45418787199ddba63eaf8c0851a1555c8a002e4c5c6220d0c8a0bba094a4f626" gracePeriod=25
Nov 02 14:33:09.182325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.182280    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:33:09.546528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.546383    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:33:09.552572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.551214    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 02 14:33:09.649162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.649120    5034 generic.go:356] "Generic (PLEG): container finished" podID="a4143f67-1e77-448f-a1a6-4e0887be109b" containerID="45418787199ddba63eaf8c0851a1555c8a002e4c5c6220d0c8a0bba094a4f626" exitCode=0
Nov 02 14:33:09.651631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.651596    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd" event={"ID":"a4143f67-1e77-448f-a1a6-4e0887be109b","Type":"ContainerDied","Data":"45418787199ddba63eaf8c0851a1555c8a002e4c5c6220d0c8a0bba094a4f626"}
Nov 02 14:33:09.653931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.653908    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:33:09.735836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.735253    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"]
Nov 02 14:33:09.738859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.738828    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:09.767863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.767814    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"]
Nov 02 14:33:09.770913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.769270    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a4143f67-1e77-448f-a1a6-4e0887be109b" containerName="payload-processing"
Nov 02 14:33:09.770913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.769296    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4143f67-1e77-448f-a1a6-4e0887be109b" containerName="payload-processing"
Nov 02 14:33:09.770913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.769609    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4143f67-1e77-448f-a1a6-4e0887be109b" containerName="payload-processing"
Nov 02 14:33:09.775804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.775064    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:09.800864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.797299    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"]
Nov 02 14:33:09.981811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:09.947842    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-plugins-config-volume\") pod \"a4143f67-1e77-448f-a1a6-4e0887be109b\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") "
Nov 02 14:33:10.051860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.012067    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-provider-credentials\") pod \"a4143f67-1e77-448f-a1a6-4e0887be109b\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") "
Nov 02 14:33:10.051860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.012185    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-kube-api-access-9nbhw\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-kube-api-access-9nbhw\") pod \"a4143f67-1e77-448f-a1a6-4e0887be109b\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") "
Nov 02 14:33:10.051860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.012287    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-routing-overlay-volume\") pod \"a4143f67-1e77-448f-a1a6-4e0887be109b\" (UID: \"a4143f67-1e77-448f-a1a6-4e0887be109b\") "
Nov 02 14:33:10.051860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.013418    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnlzd\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-kube-api-access-dnlzd\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.051860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.013693    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-routing-overlay-volume\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.051860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.013994    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-plugins-config-volume\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.051860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.014054    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-provider-credentials\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.051860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.011947    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-plugins-config-volume" pod "a4143f67-1e77-448f-a1a6-4e0887be109b" (UID: "a4143f67-1e77-448f-a1a6-4e0887be109b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:10.077798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.076480    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-routing-overlay-volume" pod "a4143f67-1e77-448f-a1a6-4e0887be109b" (UID: "a4143f67-1e77-448f-a1a6-4e0887be109b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.112385    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-kube-api-access-9nbhw" pod "a4143f67-1e77-448f-a1a6-4e0887be109b" (UID: "a4143f67-1e77-448f-a1a6-4e0887be109b"). InnerVolumeSpecName "kube-api-access-9nbhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.122911    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-plugins-config-volume\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.123085    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-provider-credentials\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.123238    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnlzd\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-kube-api-access-dnlzd\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.123463    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-routing-overlay-volume\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.123757    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.123829    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nbhw\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-kube-api-access-9nbhw\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.123870    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4143f67-1e77-448f-a1a6-4e0887be109b-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.124654    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-routing-overlay-volume\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.131859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.125247    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-plugins-config-volume\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.136563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.136536    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-provider-credentials\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.156203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.148123    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-provider-credentials" pod "a4143f67-1e77-448f-a1a6-4e0887be109b" (UID: "a4143f67-1e77-448f-a1a6-4e0887be109b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:10.159708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.159679    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnlzd\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-kube-api-access-dnlzd\") pod \"payload-processing-external-model-5698cdb5c5-9nb66\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") " pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.232806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.232263    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4143f67-1e77-448f-a1a6-4e0887be109b-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:10.410853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.410209    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:10.757812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.757436    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd" event={"ID":"a4143f67-1e77-448f-a1a6-4e0887be109b","Type":"ContainerDied","Data":"b5483fe84ff6578251b54cc96a7ab2c391526bccf4f68bdeb7d635090c5722ee"}
Nov 02 14:33:10.757812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.757510    5034 scope.go:122] "RemoveContainer" containerID="45418787199ddba63eaf8c0851a1555c8a002e4c5c6220d0c8a0bba094a4f626"
Nov 02 14:33:10.757812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.757692    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"
Nov 02 14:33:10.961034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.939634    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"]
Nov 02 14:33:10.961860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:10.961806    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-slhbd"]
Nov 02 14:33:11.619696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:11.619650    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"]
Nov 02 14:33:12.479406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:12.479370    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4143f67-1e77-448f-a1a6-4e0887be109b" path="/var/lib/kubelet/pods/a4143f67-1e77-448f-a1a6-4e0887be109b/volumes"
Nov 02 14:33:12.792816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:12.778425    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e4d71d-tv74n"]
Nov 02 14:33:12.900803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:12.885504    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"]
Nov 02 14:33:13.407865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:13.407832    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76da187f-6f50-4923-8760-a158d597e841" path="/var/lib/kubelet/pods/76da187f-6f50-4923-8760-a158d597e841/volumes"
Nov 02 14:33:13.488831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:13.488791    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66" event={"ID":"8c86bbed-1a24-46a1-9ece-8d624c9404e1","Type":"ContainerStarted","Data":"9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176"}
Nov 02 14:33:13.489405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:13.489376    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:13.489558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:13.489420    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66" event={"ID":"8c86bbed-1a24-46a1-9ece-8d624c9404e1","Type":"ContainerStarted","Data":"11f2e3b40d665a2d734816e4d57b276a35a94fb8dd45dc7e5fc35bc41335d4ec"}
Nov 02 14:33:13.533608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:13.533554    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66" podStartSLOduration=4.533539811 podStartE2EDuration="4.533539811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:13.533125939 +0000 UTC m=+2984.284430378" watchObservedRunningTime="2026-11-02 14:33:13.533539811 +0000 UTC m=+2984.284844248"
Nov 02 14:33:14.301836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.301800    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"]
Nov 02 14:33:14.308848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.308811    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.330796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.330117    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"]
Nov 02 14:33:14.335062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.335017    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bc34db\""
Nov 02 14:33:14.335217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.335051    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"agc-cred-8c6ef24cb8d4\""
Nov 02 14:33:14.335217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.335114    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"kube-root-ca.crt\""
Nov 02 14:33:14.335217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.335177    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"openshift-service-ca.crt\""
Nov 02 14:33:14.335408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.335232    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"routing-overlay\""
Nov 02 14:33:14.346885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.342743    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"payload-processing-external-model-e2e--2bdbc850-dockercfg-4xlsb\""
Nov 02 14:33:14.452367 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.452329    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhw5\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-kube-api-access-gfhw5\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.452575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.452377    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.452575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.452516    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.452700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.452605    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.578851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.570979    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.578851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.571091    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.578851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.571412    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhw5\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-kube-api-access-gfhw5\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.578851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.571453    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.578851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.572440    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.578851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.573186    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.611605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.611572    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.639615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.639584    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhw5\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-kube-api-access-gfhw5\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.653110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.653080    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:14.741041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:14.740999    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt" podUID="8175053c-ddb7-46dd-9970-4f3e996d3e64" containerName="payload-processing" containerID="cri-o://87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca" gracePeriod=25
Nov 02 14:33:15.069125 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:33:15.069085    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bc23a4_5574_4eae_9e59_7c1dc3adf786.slice/crio-c2ed46458074d4fbf940a6d74934af6c21002fa75977cf6c349d6036e1e127ca WatchSource:0}: Error finding container c2ed46458074d4fbf940a6d74934af6c21002fa75977cf6c349d6036e1e127ca: Status 404 returned error can't find the container with id c2ed46458074d4fbf940a6d74934af6c21002fa75977cf6c349d6036e1e127ca
Nov 02 14:33:15.074455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.074276    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"]
Nov 02 14:33:15.097107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.097082    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:15.189408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.189374    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-routing-overlay-volume\") pod \"8175053c-ddb7-46dd-9970-4f3e996d3e64\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") "
Nov 02 14:33:15.189579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.189521    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-provider-credentials\") pod \"8175053c-ddb7-46dd-9970-4f3e996d3e64\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") "
Nov 02 14:33:15.189654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.189607    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-kube-api-access-9lnv9\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-kube-api-access-9lnv9\") pod \"8175053c-ddb7-46dd-9970-4f3e996d3e64\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") "
Nov 02 14:33:15.189795 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.189758    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-routing-overlay-volume" pod "8175053c-ddb7-46dd-9970-4f3e996d3e64" (UID: "8175053c-ddb7-46dd-9970-4f3e996d3e64"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:15.189885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.189825    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-plugins-config-volume\") pod \"8175053c-ddb7-46dd-9970-4f3e996d3e64\" (UID: \"8175053c-ddb7-46dd-9970-4f3e996d3e64\") "
Nov 02 14:33:15.190366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.190337    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-plugins-config-volume" pod "8175053c-ddb7-46dd-9970-4f3e996d3e64" (UID: "8175053c-ddb7-46dd-9970-4f3e996d3e64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:15.190873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.190842    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:15.190873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.190870    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8175053c-ddb7-46dd-9970-4f3e996d3e64-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:15.193448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.193419    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-kube-api-access-9lnv9" pod "8175053c-ddb7-46dd-9970-4f3e996d3e64" (UID: "8175053c-ddb7-46dd-9970-4f3e996d3e64"). InnerVolumeSpecName "kube-api-access-9lnv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:15.193557 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.193466    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-provider-credentials" pod "8175053c-ddb7-46dd-9970-4f3e996d3e64" (UID: "8175053c-ddb7-46dd-9970-4f3e996d3e64"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:15.293211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.293173    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:15.293211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.293210    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lnv9\" (UniqueName: \"kubernetes.io/projected/8175053c-ddb7-46dd-9970-4f3e996d3e64-kube-api-access-9lnv9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:15.408845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.407411    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"]
Nov 02 14:33:15.408845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.408301    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8175053c-ddb7-46dd-9970-4f3e996d3e64" containerName="payload-processing"
Nov 02 14:33:15.408845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.408318    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8175053c-ddb7-46dd-9970-4f3e996d3e64" containerName="payload-processing"
Nov 02 14:33:15.408845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.408560    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8175053c-ddb7-46dd-9970-4f3e996d3e64" containerName="payload-processing"
Nov 02 14:33:15.413688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.413666    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"]
Nov 02 14:33:15.413853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.413832    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.454963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.447303    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"routing-overlay\""
Nov 02 14:33:15.454963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.448421    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"payload-processing-external-model-e2e--005c2a0a-dockercfg-pqmxh\""
Nov 02 14:33:15.454963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.448529    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"agc-cred-1e3a26fb0e59\""
Nov 02 14:33:15.454963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.448664    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"kube-root-ca.crt\""
Nov 02 14:33:15.454963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.448745    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"openshift-service-ca.crt\""
Nov 02 14:33:15.454963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.448844    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-87efd3\""
Nov 02 14:33:15.556801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.556377    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.556801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.556626    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.685977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.685854    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.708199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.708157    5034 generic.go:356] "Generic (PLEG): container finished" podID="8175053c-ddb7-46dd-9970-4f3e996d3e64" containerID="87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca" exitCode=0
Nov 02 14:33:15.708704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.708677    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"
Nov 02 14:33:15.713962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.713935    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt" event={"ID":"8175053c-ddb7-46dd-9970-4f3e996d3e64","Type":"ContainerDied","Data":"87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca"}
Nov 02 14:33:15.714955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.714931    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt" event={"ID":"8175053c-ddb7-46dd-9970-4f3e996d3e64","Type":"ContainerDied","Data":"b39c814e76f453beb143313a6316c315ab35986a9b712d312997bd2d5d350759"}
Nov 02 14:33:15.715118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.715100    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9" event={"ID":"d5bc23a4-5574-4eae-9e59-7c1dc3adf786","Type":"ContainerStarted","Data":"e059deec8af3f13ba12ed53683b020be50c4b60a11a2447e795c1c8d87d10209"}
Nov 02 14:33:15.715232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.715219    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9" event={"ID":"d5bc23a4-5574-4eae-9e59-7c1dc3adf786","Type":"ContainerStarted","Data":"c2ed46458074d4fbf940a6d74934af6c21002fa75977cf6c349d6036e1e127ca"}
Nov 02 14:33:15.715338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.715327    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:15.715447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.715123    5034 scope.go:122] "RemoveContainer" containerID="87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca"
Nov 02 14:33:15.715759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.692552    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfnlv\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-kube-api-access-mfnlv\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.739615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.739593    5034 scope.go:122] "RemoveContainer" containerID="87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca"
Nov 02 14:33:15.740010 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:33:15.739965    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca\": container with ID starting with 87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca not found: ID does not exist" containerID="87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca"
Nov 02 14:33:15.740138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.740024    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca"} err="failed to get container status \"87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca\": rpc error: code = NotFound desc = could not find container \"87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca\": container with ID starting with 87dc543cf2ac029765a8526bf04344ffad76816fbd1aa0b75d7250b7f94ec3ca not found: ID does not exist"
Nov 02 14:33:15.761114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.761065    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9" podStartSLOduration=1.7610506620000002 podStartE2EDuration="1.761050662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:15.759198799 +0000 UTC m=+2986.510503239" watchObservedRunningTime="2026-11-02 14:33:15.761050662 +0000 UTC m=+2986.512355100"
Nov 02 14:33:15.802895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.802581    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"]
Nov 02 14:33:15.809131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.809106    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db49ff4fd-jm7jt"]
Nov 02 14:33:15.817382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.817353    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.817529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.817403    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfnlv\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-kube-api-access-mfnlv\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.817692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.817668    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.818189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.817875    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.818624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.818598    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.818738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.818618    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.820305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.820277    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:15.843975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:15.843942    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfnlv\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-kube-api-access-mfnlv\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:16.004304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:16.004269    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:16.341793 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:33:16.337708    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab87cc3b_7de5_4e26_9241_519a4e003def.slice/crio-f810a34e0b9fa02c65555afe8ad6af2a677c63daea77168f576990c107d9a769 WatchSource:0}: Error finding container f810a34e0b9fa02c65555afe8ad6af2a677c63daea77168f576990c107d9a769: Status 404 returned error can't find the container with id f810a34e0b9fa02c65555afe8ad6af2a677c63daea77168f576990c107d9a769
Nov 02 14:33:16.341793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:16.340556    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"]
Nov 02 14:33:16.727711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:16.727616    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4" event={"ID":"ab87cc3b-7de5-4e26-9241-519a4e003def","Type":"ContainerStarted","Data":"e50c9ca7a5c421fd53070d28ee90929c1eb76d29c36ae1af235d1580b2c5f525"}
Nov 02 14:33:16.727711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:16.727680    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4" event={"ID":"ab87cc3b-7de5-4e26-9241-519a4e003def","Type":"ContainerStarted","Data":"f810a34e0b9fa02c65555afe8ad6af2a677c63daea77168f576990c107d9a769"}
Nov 02 14:33:16.728294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:16.727713    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:16.776757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:16.776684    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4" podStartSLOduration=1.776660187 podStartE2EDuration="1.776660187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:16.775111464 +0000 UTC m=+2987.526415902" watchObservedRunningTime="2026-11-02 14:33:16.776660187 +0000 UTC m=+2987.527964652"
Nov 02 14:33:17.401582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:17.401549    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8175053c-ddb7-46dd-9970-4f3e996d3e64" path="/var/lib/kubelet/pods/8175053c-ddb7-46dd-9970-4f3e996d3e64/volumes"
Nov 02 14:33:18.415201 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:33:18.415165    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:33:24.617065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:24.617028    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:33:26.731526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:26.731497    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:27.176110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.176023    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"]
Nov 02 14:33:27.183905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.183880    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.200993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.200968    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"]
Nov 02 14:33:27.202947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.202925    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-87efd3\"/\"agc-cred-b174148781eb\""
Nov 02 14:33:27.249841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.249761    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.249995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.249966    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfprj\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-kube-api-access-jfprj\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.250088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.250031    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.250267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.250241    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.352063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.352035    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.352241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.352103    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.352241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.352207    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfprj\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-kube-api-access-jfprj\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.352375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.352263    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.352927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.352901    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.353061 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.353036    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.354910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.354886    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.376601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.376578    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfprj\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-kube-api-access-jfprj\") pod \"payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") " pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.503661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.503634    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:27.741005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.740978    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:27.818608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.818571    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"]
Nov 02 14:33:27.823542 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:33:27.823509    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc067926_6d67_4ff5_aca8_666478c5fc6f.slice/crio-3c22a45342a77abc254f4738a33b3d56fdac5900003f438eeb992c95b2855e8d WatchSource:0}: Error finding container 3c22a45342a77abc254f4738a33b3d56fdac5900003f438eeb992c95b2855e8d: Status 404 returned error can't find the container with id 3c22a45342a77abc254f4738a33b3d56fdac5900003f438eeb992c95b2855e8d
Nov 02 14:33:27.880638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.880605    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" event={"ID":"fc067926-6d67-4ff5-aca8-666478c5fc6f","Type":"ContainerStarted","Data":"3c22a45342a77abc254f4738a33b3d56fdac5900003f438eeb992c95b2855e8d"}
Nov 02 14:33:27.970211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.970184    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"]
Nov 02 14:33:27.976837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.976815    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:27.994812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:27.993555    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bc34db\"/\"agc-cred-1a540b4b24e4\""
Nov 02 14:33:28.010705 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.010673    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"]
Nov 02 14:33:28.063109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.063014    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.063109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.063083    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrbf\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-kube-api-access-5lrbf\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.063423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.063396    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.063513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.063456    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.165383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.165352    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.165565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.165398    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrbf\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-kube-api-access-5lrbf\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.165627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.165586    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.165627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.165616    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.166459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.166436    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.166567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.166504    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.167952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.167927    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.190290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.190266    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrbf\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-kube-api-access-5lrbf\") pod \"payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") " pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.292643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.292601    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.610661 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:33:28.610630    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591b724a_9b26_41d7_8085_eeab89e67d45.slice/crio-153d7ea7ed917f3571d69d99ead9c793e34fd3b1957666db102b969f293e7c0b WatchSource:0}: Error finding container 153d7ea7ed917f3571d69d99ead9c793e34fd3b1957666db102b969f293e7c0b: Status 404 returned error can't find the container with id 153d7ea7ed917f3571d69d99ead9c793e34fd3b1957666db102b969f293e7c0b
Nov 02 14:33:28.613227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.613202    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"]
Nov 02 14:33:28.895334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.895223    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" event={"ID":"fc067926-6d67-4ff5-aca8-666478c5fc6f","Type":"ContainerStarted","Data":"45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5"}
Nov 02 14:33:28.895334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.895314    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:28.897411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.897379    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm" event={"ID":"591b724a-9b26-41d7-8085-eeab89e67d45","Type":"ContainerStarted","Data":"071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec"}
Nov 02 14:33:28.897535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.897440    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:28.897535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.897459    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm" event={"ID":"591b724a-9b26-41d7-8085-eeab89e67d45","Type":"ContainerStarted","Data":"153d7ea7ed917f3571d69d99ead9c793e34fd3b1957666db102b969f293e7c0b"}
Nov 02 14:33:28.943625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.943574    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" podStartSLOduration=1.943560747 podStartE2EDuration="1.943560747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:28.939592333 +0000 UTC m=+2999.690896772" watchObservedRunningTime="2026-11-02 14:33:28.943560747 +0000 UTC m=+2999.694865199"
Nov 02 14:33:28.986679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:28.986624    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm" podStartSLOduration=1.986607261 podStartE2EDuration="1.986607261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:33:28.984933556 +0000 UTC m=+2999.736237994" watchObservedRunningTime="2026-11-02 14:33:28.986607261 +0000 UTC m=+2999.737911699"
Nov 02 14:33:30.701525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:30.701490    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:33:30.702055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:30.701665    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:33:37.429189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:37.429157    5034 scope.go:122] "RemoveContainer" containerID="2f05f7b153dcab8c6ef265acadfa2b3d75710466ebcce15bf4598b4a18520cf2"
Nov 02 14:33:37.440886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:37.440858    5034 scope.go:122] "RemoveContainer" containerID="32a27f1e17b312764fef70b93c75e5a3e46af473bcd15128063525d03d835d27"
Nov 02 14:33:37.452083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:37.452056    5034 scope.go:122] "RemoveContainer" containerID="600e0c0709cfd92adbf467f3c8dc867d24dfbddf106c06ad59f1004641bd6073"
Nov 02 14:33:37.462069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:37.462045    5034 scope.go:122] "RemoveContainer" containerID="d710f3adf4e31da8445305c6005848bdd34daba2baf7c4eacf95b6ff7188d8c7"
Nov 02 14:33:37.473556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:37.473536    5034 scope.go:122] "RemoveContainer" containerID="ccf65bb76946ffeb8da05693861129aefd37e8d1475f65d19f660a878439a088"
Nov 02 14:33:39.911639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:39.911609    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:33:39.912139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:39.911685    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:33:40.011313 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:40.011273    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"]
Nov 02 14:33:40.052771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:40.052736    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"]
Nov 02 14:33:45.013119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.013002    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4" podUID="ab87cc3b-7de5-4e26-9241-519a4e003def" containerName="payload-processing" containerID="cri-o://e50c9ca7a5c421fd53070d28ee90929c1eb76d29c36ae1af235d1580b2c5f525" gracePeriod=25
Nov 02 14:33:45.053513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.053478    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9" podUID="d5bc23a4-5574-4eae-9e59-7c1dc3adf786" containerName="payload-processing" containerID="cri-o://e059deec8af3f13ba12ed53683b020be50c4b60a11a2447e795c1c8d87d10209" gracePeriod=25
Nov 02 14:33:45.190724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.190689    5034 generic.go:356] "Generic (PLEG): container finished" podID="ab87cc3b-7de5-4e26-9241-519a4e003def" containerID="e50c9ca7a5c421fd53070d28ee90929c1eb76d29c36ae1af235d1580b2c5f525" exitCode=0
Nov 02 14:33:45.190973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.190853    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4" event={"ID":"ab87cc3b-7de5-4e26-9241-519a4e003def","Type":"ContainerDied","Data":"e50c9ca7a5c421fd53070d28ee90929c1eb76d29c36ae1af235d1580b2c5f525"}
Nov 02 14:33:45.192890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.192863    5034 generic.go:356] "Generic (PLEG): container finished" podID="d5bc23a4-5574-4eae-9e59-7c1dc3adf786" containerID="e059deec8af3f13ba12ed53683b020be50c4b60a11a2447e795c1c8d87d10209" exitCode=0
Nov 02 14:33:45.193042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.192944    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9" event={"ID":"d5bc23a4-5574-4eae-9e59-7c1dc3adf786","Type":"ContainerDied","Data":"e059deec8af3f13ba12ed53683b020be50c4b60a11a2447e795c1c8d87d10209"}
Nov 02 14:33:45.414200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.414168    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:45.416345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.416287    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:45.544853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.544826    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-routing-overlay-volume\") pod \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") "
Nov 02 14:33:45.545037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.544860    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-routing-overlay-volume\") pod \"ab87cc3b-7de5-4e26-9241-519a4e003def\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") "
Nov 02 14:33:45.545037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.544927    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-provider-credentials\") pod \"ab87cc3b-7de5-4e26-9241-519a4e003def\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") "
Nov 02 14:33:45.545224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.545081    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-plugins-config-volume\") pod \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") "
Nov 02 14:33:45.545224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.545163    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-kube-api-access-gfhw5\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-kube-api-access-gfhw5\") pod \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") "
Nov 02 14:33:45.545353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.545253    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-kube-api-access-mfnlv\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-kube-api-access-mfnlv\") pod \"ab87cc3b-7de5-4e26-9241-519a4e003def\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") "
Nov 02 14:33:45.545353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.545256    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-routing-overlay-volume" pod "d5bc23a4-5574-4eae-9e59-7c1dc3adf786" (UID: "d5bc23a4-5574-4eae-9e59-7c1dc3adf786"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:45.545469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.545423    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-plugins-config-volume\") pod \"ab87cc3b-7de5-4e26-9241-519a4e003def\" (UID: \"ab87cc3b-7de5-4e26-9241-519a4e003def\") "
Nov 02 14:33:45.545529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.545487    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-provider-credentials\") pod \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\" (UID: \"d5bc23a4-5574-4eae-9e59-7c1dc3adf786\") "
Nov 02 14:33:45.545962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.545714    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-routing-overlay-volume" pod "ab87cc3b-7de5-4e26-9241-519a4e003def" (UID: "ab87cc3b-7de5-4e26-9241-519a4e003def"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:45.545962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.545726    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-plugins-config-volume" pod "d5bc23a4-5574-4eae-9e59-7c1dc3adf786" (UID: "d5bc23a4-5574-4eae-9e59-7c1dc3adf786"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:45.545962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.545879    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-plugins-config-volume" pod "ab87cc3b-7de5-4e26-9241-519a4e003def" (UID: "ab87cc3b-7de5-4e26-9241-519a4e003def"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:33:45.546440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.546415    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:45.546534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.546447    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:45.546534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.546463    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:45.546534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.546477    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab87cc3b-7de5-4e26-9241-519a4e003def-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:45.547945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.547922    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-kube-api-access-gfhw5" pod "d5bc23a4-5574-4eae-9e59-7c1dc3adf786" (UID: "d5bc23a4-5574-4eae-9e59-7c1dc3adf786"). InnerVolumeSpecName "kube-api-access-gfhw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:45.548283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.548262    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-provider-credentials" pod "d5bc23a4-5574-4eae-9e59-7c1dc3adf786" (UID: "d5bc23a4-5574-4eae-9e59-7c1dc3adf786"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:45.548363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.548307    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-kube-api-access-mfnlv" pod "ab87cc3b-7de5-4e26-9241-519a4e003def" (UID: "ab87cc3b-7de5-4e26-9241-519a4e003def"). InnerVolumeSpecName "kube-api-access-mfnlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:45.548363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.548341    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-provider-credentials" pod "ab87cc3b-7de5-4e26-9241-519a4e003def" (UID: "ab87cc3b-7de5-4e26-9241-519a4e003def"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:33:45.647618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.647533    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:45.647618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.647567    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:45.647618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.647577    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfhw5\" (UniqueName: \"kubernetes.io/projected/d5bc23a4-5574-4eae-9e59-7c1dc3adf786-kube-api-access-gfhw5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:45.647618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:45.647586    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfnlv\" (UniqueName: \"kubernetes.io/projected/ab87cc3b-7de5-4e26-9241-519a4e003def-kube-api-access-mfnlv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:33:46.204172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.204138    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"
Nov 02 14:33:46.204172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.204161    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4" event={"ID":"ab87cc3b-7de5-4e26-9241-519a4e003def","Type":"ContainerDied","Data":"f810a34e0b9fa02c65555afe8ad6af2a677c63daea77168f576990c107d9a769"}
Nov 02 14:33:46.204658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.204227    5034 scope.go:122] "RemoveContainer" containerID="e50c9ca7a5c421fd53070d28ee90929c1eb76d29c36ae1af235d1580b2c5f525"
Nov 02 14:33:46.207879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.207850    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"
Nov 02 14:33:46.207879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.207866    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9" event={"ID":"d5bc23a4-5574-4eae-9e59-7c1dc3adf786","Type":"ContainerDied","Data":"c2ed46458074d4fbf940a6d74934af6c21002fa75977cf6c349d6036e1e127ca"}
Nov 02 14:33:46.222936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.222905    5034 scope.go:122] "RemoveContainer" containerID="e059deec8af3f13ba12ed53683b020be50c4b60a11a2447e795c1c8d87d10209"
Nov 02 14:33:46.286844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.286812    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"]
Nov 02 14:33:46.299109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.299050    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-669f69px4"]
Nov 02 14:33:46.328299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.328271    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"]
Nov 02 14:33:46.332893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:46.332867    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-64cd5dnr9"]
Nov 02 14:33:47.398569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:47.398537    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab87cc3b-7de5-4e26-9241-519a4e003def" path="/var/lib/kubelet/pods/ab87cc3b-7de5-4e26-9241-519a4e003def/volumes"
Nov 02 14:33:47.399025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:33:47.399017    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5bc23a4-5574-4eae-9e59-7c1dc3adf786" path="/var/lib/kubelet/pods/d5bc23a4-5574-4eae-9e59-7c1dc3adf786/volumes"
Nov 02 14:33:48.415973 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:33:48.415932    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:34:06.053065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:06.052988    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"]
Nov 02 14:34:06.064864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:06.064830    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5abfc98-64xbh"]
Nov 02 14:34:07.454721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:07.454685    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="774ec3a9-cda4-4ea3-a028-c23476678fdb" path="/var/lib/kubelet/pods/774ec3a9-cda4-4ea3-a028-c23476678fdb/volumes"
Nov 02 14:34:11.747088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.747047    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"]
Nov 02 14:34:11.748410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.748377    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d5bc23a4-5574-4eae-9e59-7c1dc3adf786" containerName="payload-processing"
Nov 02 14:34:11.748410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.748408    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bc23a4-5574-4eae-9e59-7c1dc3adf786" containerName="payload-processing"
Nov 02 14:34:11.748593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.748524    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ab87cc3b-7de5-4e26-9241-519a4e003def" containerName="payload-processing"
Nov 02 14:34:11.748593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.748533    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab87cc3b-7de5-4e26-9241-519a4e003def" containerName="payload-processing"
Nov 02 14:34:11.748758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.748742    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bc23a4-5574-4eae-9e59-7c1dc3adf786" containerName="payload-processing"
Nov 02 14:34:11.748834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.748801    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab87cc3b-7de5-4e26-9241-519a4e003def" containerName="payload-processing"
Nov 02 14:34:11.754138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.754117    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:11.771835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.771807    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"]
Nov 02 14:34:11.776179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.776153    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:34:11.776179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.776159    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:34:11.776345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.776188    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:34:11.783840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.783818    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:34:11.899349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.899310    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpqk\" (UniqueName: \"kubernetes.io/projected/acd28dcd-1c18-43ab-924d-0abbb0df7789-kube-api-access-xhpqk\") pod \"maas-api-revoke-sub-ce4cb093-9vs6c\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:11.899349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.899356    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce4cb093-9vs6c\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:11.899634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:11.899417    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce4cb093-9vs6c\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:12.001803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.000853    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpqk\" (UniqueName: \"kubernetes.io/projected/acd28dcd-1c18-43ab-924d-0abbb0df7789-kube-api-access-xhpqk\") pod \"maas-api-revoke-sub-ce4cb093-9vs6c\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:12.001803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.000910    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce4cb093-9vs6c\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:12.001803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.001006    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce4cb093-9vs6c\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:12.003892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.002579    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce4cb093-9vs6c\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:12.004225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.004202    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce4cb093-9vs6c\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:12.031026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.030998    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpqk\" (UniqueName: \"kubernetes.io/projected/acd28dcd-1c18-43ab-924d-0abbb0df7789-kube-api-access-xhpqk\") pod \"maas-api-revoke-sub-ce4cb093-9vs6c\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:12.072477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.072447    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:12.391093 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:34:12.391051    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd28dcd_1c18_43ab_924d_0abbb0df7789.slice/crio-337872f542ba590c43e74d8897f2c56e6da30d3ee09474510bcb2341a6799dcf WatchSource:0}: Error finding container 337872f542ba590c43e74d8897f2c56e6da30d3ee09474510bcb2341a6799dcf: Status 404 returned error can't find the container with id 337872f542ba590c43e74d8897f2c56e6da30d3ee09474510bcb2341a6799dcf
Nov 02 14:34:12.392859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.392835    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"]
Nov 02 14:34:12.564999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.564967    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c" event={"ID":"acd28dcd-1c18-43ab-924d-0abbb0df7789","Type":"ContainerStarted","Data":"337872f542ba590c43e74d8897f2c56e6da30d3ee09474510bcb2341a6799dcf"}
Nov 02 14:34:12.821148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.814814    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"]
Nov 02 14:34:12.832306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.832284    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:12.834246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.832185    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"]
Nov 02 14:34:12.916146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.916115    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc09e0ea-8qgg6\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:12.916331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.916302    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b6t4\" (UniqueName: \"kubernetes.io/projected/f111d75b-bba3-426f-b366-f471654ae09f-kube-api-access-6b6t4\") pod \"maas-api-revoke-sub-cc09e0ea-8qgg6\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:12.916462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:12.916448    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc09e0ea-8qgg6\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:13.017500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.017470    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc09e0ea-8qgg6\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:13.017696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.017652    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b6t4\" (UniqueName: \"kubernetes.io/projected/f111d75b-bba3-426f-b366-f471654ae09f-kube-api-access-6b6t4\") pod \"maas-api-revoke-sub-cc09e0ea-8qgg6\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:13.018032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.017809    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc09e0ea-8qgg6\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:13.018312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.018291    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc09e0ea-8qgg6\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:13.020386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.020363    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc09e0ea-8qgg6\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:13.042926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.042905    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b6t4\" (UniqueName: \"kubernetes.io/projected/f111d75b-bba3-426f-b366-f471654ae09f-kube-api-access-6b6t4\") pod \"maas-api-revoke-sub-cc09e0ea-8qgg6\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:13.155063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.154986    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:13.473464 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:34:13.473430    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf111d75b_bba3_426f_b366_f471654ae09f.slice/crio-92b431991b1dd68eca709ee2609f528d88c395912c197e9c24feb4df9952c462 WatchSource:0}: Error finding container 92b431991b1dd68eca709ee2609f528d88c395912c197e9c24feb4df9952c462: Status 404 returned error can't find the container with id 92b431991b1dd68eca709ee2609f528d88c395912c197e9c24feb4df9952c462
Nov 02 14:34:13.475403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.475377    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"]
Nov 02 14:34:13.577168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.577139    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6" event={"ID":"f111d75b-bba3-426f-b366-f471654ae09f","Type":"ContainerStarted","Data":"92b431991b1dd68eca709ee2609f528d88c395912c197e9c24feb4df9952c462"}
Nov 02 14:34:13.579100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.579075    5034 generic.go:356] "Generic (PLEG): container finished" podID="acd28dcd-1c18-43ab-924d-0abbb0df7789" containerID="1eaf8a35aaf1b137b4d35626b4a1a1396af6e229c5e547dc4df2e2d0851b4c15" exitCode=0
Nov 02 14:34:13.579224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:13.579159    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c" event={"ID":"acd28dcd-1c18-43ab-924d-0abbb0df7789","Type":"ContainerDied","Data":"1eaf8a35aaf1b137b4d35626b4a1a1396af6e229c5e547dc4df2e2d0851b4c15"}
Nov 02 14:34:14.591535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.591495    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6" event={"ID":"f111d75b-bba3-426f-b366-f471654ae09f","Type":"ContainerStarted","Data":"f0a127bf3afe8822ff556b82b095a250aac03802a81a83f866647f303dabc7c5"}
Nov 02 14:34:14.636078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.636024    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6" podStartSLOduration=2.636009714 podStartE2EDuration="2.636009714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:34:14.634796012 +0000 UTC m=+3045.386100441" watchObservedRunningTime="2026-11-02 14:34:14.636009714 +0000 UTC m=+3045.387314151"
Nov 02 14:34:14.778869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.778842    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:14.850826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.850705    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-xks-ca\") pod \"acd28dcd-1c18-43ab-924d-0abbb0df7789\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") "
Nov 02 14:34:14.850999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.850937    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd28dcd-1c18-43ab-924d-0abbb0df7789-kube-api-access-xhpqk\" (UniqueName: \"kubernetes.io/projected/acd28dcd-1c18-43ab-924d-0abbb0df7789-kube-api-access-xhpqk\") pod \"acd28dcd-1c18-43ab-924d-0abbb0df7789\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") "
Nov 02 14:34:14.851090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.851010    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-openshift-ca\") pod \"acd28dcd-1c18-43ab-924d-0abbb0df7789\" (UID: \"acd28dcd-1c18-43ab-924d-0abbb0df7789\") "
Nov 02 14:34:14.852212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.852184    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-openshift-ca" pod "acd28dcd-1c18-43ab-924d-0abbb0df7789" (UID: "acd28dcd-1c18-43ab-924d-0abbb0df7789"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:34:14.855847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.855822    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-xks-ca" pod "acd28dcd-1c18-43ab-924d-0abbb0df7789" (UID: "acd28dcd-1c18-43ab-924d-0abbb0df7789"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:34:14.858988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.858961    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd28dcd-1c18-43ab-924d-0abbb0df7789-kube-api-access-xhpqk" pod "acd28dcd-1c18-43ab-924d-0abbb0df7789" (UID: "acd28dcd-1c18-43ab-924d-0abbb0df7789"). InnerVolumeSpecName "kube-api-access-xhpqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:34:14.954052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.954023    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:14.954052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.954049    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acd28dcd-1c18-43ab-924d-0abbb0df7789-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:14.954052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:14.954058    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhpqk\" (UniqueName: \"kubernetes.io/projected/acd28dcd-1c18-43ab-924d-0abbb0df7789-kube-api-access-xhpqk\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:15.603075 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:15.603037    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"
Nov 02 14:34:15.603521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:15.603074    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c" event={"ID":"acd28dcd-1c18-43ab-924d-0abbb0df7789","Type":"ContainerDied","Data":"337872f542ba590c43e74d8897f2c56e6da30d3ee09474510bcb2341a6799dcf"}
Nov 02 14:34:15.603521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:15.603131    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="337872f542ba590c43e74d8897f2c56e6da30d3ee09474510bcb2341a6799dcf"
Nov 02 14:34:15.605226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:15.605194    5034 generic.go:356] "Generic (PLEG): container finished" podID="f111d75b-bba3-426f-b366-f471654ae09f" containerID="f0a127bf3afe8822ff556b82b095a250aac03802a81a83f866647f303dabc7c5" exitCode=0
Nov 02 14:34:15.605392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:15.605373    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6" event={"ID":"f111d75b-bba3-426f-b366-f471654ae09f","Type":"ContainerDied","Data":"f0a127bf3afe8822ff556b82b095a250aac03802a81a83f866647f303dabc7c5"}
Nov 02 14:34:16.790507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.790481    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:16.887126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.887022    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f111d75b-bba3-426f-b366-f471654ae09f-kube-api-access-6b6t4\" (UniqueName: \"kubernetes.io/projected/f111d75b-bba3-426f-b366-f471654ae09f-kube-api-access-6b6t4\") pod \"f111d75b-bba3-426f-b366-f471654ae09f\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") "
Nov 02 14:34:16.887450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.887423    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-openshift-ca\") pod \"f111d75b-bba3-426f-b366-f471654ae09f\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") "
Nov 02 14:34:16.887617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.887534    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-xks-ca\") pod \"f111d75b-bba3-426f-b366-f471654ae09f\" (UID: \"f111d75b-bba3-426f-b366-f471654ae09f\") "
Nov 02 14:34:16.887869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.887838    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-openshift-ca" pod "f111d75b-bba3-426f-b366-f471654ae09f" (UID: "f111d75b-bba3-426f-b366-f471654ae09f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:34:16.888372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.888346    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:16.890055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.890006    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f111d75b-bba3-426f-b366-f471654ae09f-kube-api-access-6b6t4" pod "f111d75b-bba3-426f-b366-f471654ae09f" (UID: "f111d75b-bba3-426f-b366-f471654ae09f"). InnerVolumeSpecName "kube-api-access-6b6t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:34:16.890206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.890180    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-xks-ca" pod "f111d75b-bba3-426f-b366-f471654ae09f" (UID: "f111d75b-bba3-426f-b366-f471654ae09f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:34:16.990316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.990281    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b6t4\" (UniqueName: \"kubernetes.io/projected/f111d75b-bba3-426f-b366-f471654ae09f-kube-api-access-6b6t4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:16.990316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:16.990311    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f111d75b-bba3-426f-b366-f471654ae09f-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:17.626949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:17.626915    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"
Nov 02 14:34:17.627128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:17.626952    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6" event={"ID":"f111d75b-bba3-426f-b366-f471654ae09f","Type":"ContainerDied","Data":"92b431991b1dd68eca709ee2609f528d88c395912c197e9c24feb4df9952c462"}
Nov 02 14:34:17.627128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:17.627000    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92b431991b1dd68eca709ee2609f528d88c395912c197e9c24feb4df9952c462"
Nov 02 14:34:18.416798 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:34:18.416746    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:34:22.031151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:22.031116    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"]
Nov 02 14:34:22.044177 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:22.044113    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd02c09-sczk8"]
Nov 02 14:34:22.929523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:22.929489    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"]
Nov 02 14:34:23.505802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:23.498131    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8b7229-44eb-4f23-b967-24badcb201d8" path="/var/lib/kubelet/pods/9d8b7229-44eb-4f23-b967-24badcb201d8/volumes"
Nov 02 14:34:25.421833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.419812    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"]
Nov 02 14:34:25.421833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.421278    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f111d75b-bba3-426f-b366-f471654ae09f" containerName="revoke-keys"
Nov 02 14:34:25.421833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.421301    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f111d75b-bba3-426f-b366-f471654ae09f" containerName="revoke-keys"
Nov 02 14:34:25.421833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.421321    5034 cpu_manager.go:414] "containerMap: removing container" podUID="acd28dcd-1c18-43ab-924d-0abbb0df7789" containerName="revoke-keys"
Nov 02 14:34:25.421833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.421328    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd28dcd-1c18-43ab-924d-0abbb0df7789" containerName="revoke-keys"
Nov 02 14:34:25.421833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.421552    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f111d75b-bba3-426f-b366-f471654ae09f" containerName="revoke-keys"
Nov 02 14:34:25.421833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.421564    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd28dcd-1c18-43ab-924d-0abbb0df7789" containerName="revoke-keys"
Nov 02 14:34:25.427719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.427687    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:25.450654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.450603    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"]
Nov 02 14:34:25.455086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.454945    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:34:25.455086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.455000    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:34:25.455086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.455060    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:34:25.462116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.462077    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:34:25.578414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.578377    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljhx\" (UniqueName: \"kubernetes.io/projected/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-kube-api-access-8ljhx\") pod \"maas-api-revoke-sub-248b3f02-6wn4p\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:25.578706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.578688    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248b3f02-6wn4p\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:25.579094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.579075    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248b3f02-6wn4p\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:25.686878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.680916    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljhx\" (UniqueName: \"kubernetes.io/projected/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-kube-api-access-8ljhx\") pod \"maas-api-revoke-sub-248b3f02-6wn4p\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:25.686878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.680982    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248b3f02-6wn4p\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:25.686878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.681190    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248b3f02-6wn4p\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:25.686878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.682189    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-248b3f02-6wn4p\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:25.880515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.879814    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljhx\" (UniqueName: \"kubernetes.io/projected/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-kube-api-access-8ljhx\") pod \"maas-api-revoke-sub-248b3f02-6wn4p\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:25.910809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:25.909246    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-248b3f02-6wn4p\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:26.054423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:26.054394    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:26.406269 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:34:26.406225    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93c95ca_53f4_4076_98b9_c156cb0d7d9b.slice/crio-26a224581bef7221fd97bc92cee2da046d82c4f62ec66b952185d06451ea61ca WatchSource:0}: Error finding container 26a224581bef7221fd97bc92cee2da046d82c4f62ec66b952185d06451ea61ca: Status 404 returned error can't find the container with id 26a224581bef7221fd97bc92cee2da046d82c4f62ec66b952185d06451ea61ca
Nov 02 14:34:26.411695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:26.411663    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"]
Nov 02 14:34:26.838869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:26.838833    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p" event={"ID":"b93c95ca-53f4-4076-98b9-c156cb0d7d9b","Type":"ContainerStarted","Data":"26a224581bef7221fd97bc92cee2da046d82c4f62ec66b952185d06451ea61ca"}
Nov 02 14:34:27.851824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:27.851704    5034 generic.go:356] "Generic (PLEG): container finished" podID="b93c95ca-53f4-4076-98b9-c156cb0d7d9b" containerID="85cbd5235a0ff15ec84172a39d583f158bb396ba1f1369f64dc27284bcb95236" exitCode=0
Nov 02 14:34:27.852222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:27.851822    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p" event={"ID":"b93c95ca-53f4-4076-98b9-c156cb0d7d9b","Type":"ContainerDied","Data":"85cbd5235a0ff15ec84172a39d583f158bb396ba1f1369f64dc27284bcb95236"}
Nov 02 14:34:27.930475 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:27.930423    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66" podUID="8c86bbed-1a24-46a1-9ece-8d624c9404e1" containerName="payload-processing" containerID="cri-o://9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176" gracePeriod=25
Nov 02 14:34:28.250524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.250494    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:34:28.362348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.362239    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-plugins-config-volume\") pod \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") "
Nov 02 14:34:28.362829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.362755    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-kube-api-access-dnlzd\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-kube-api-access-dnlzd\") pod \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") "
Nov 02 14:34:28.363341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.363293    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-provider-credentials\") pod \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") "
Nov 02 14:34:28.363557 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.363529    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-routing-overlay-volume\") pod \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\" (UID: \"8c86bbed-1a24-46a1-9ece-8d624c9404e1\") "
Nov 02 14:34:28.364939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.364916    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-routing-overlay-volume" pod "8c86bbed-1a24-46a1-9ece-8d624c9404e1" (UID: "8c86bbed-1a24-46a1-9ece-8d624c9404e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:34:28.365598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.365578    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-plugins-config-volume" pod "8c86bbed-1a24-46a1-9ece-8d624c9404e1" (UID: "8c86bbed-1a24-46a1-9ece-8d624c9404e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:34:28.380443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.372730    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-kube-api-access-dnlzd" pod "8c86bbed-1a24-46a1-9ece-8d624c9404e1" (UID: "8c86bbed-1a24-46a1-9ece-8d624c9404e1"). InnerVolumeSpecName "kube-api-access-dnlzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:34:28.390740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.390711    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-provider-credentials" pod "8c86bbed-1a24-46a1-9ece-8d624c9404e1" (UID: "8c86bbed-1a24-46a1-9ece-8d624c9404e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:34:28.466201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.466174    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:28.466201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.466201    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:28.466372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.466210    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c86bbed-1a24-46a1-9ece-8d624c9404e1-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:28.466372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.466220    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnlzd\" (UniqueName: \"kubernetes.io/projected/8c86bbed-1a24-46a1-9ece-8d624c9404e1-kube-api-access-dnlzd\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:28.861934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.861895    5034 generic.go:356] "Generic (PLEG): container finished" podID="8c86bbed-1a24-46a1-9ece-8d624c9404e1" containerID="9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176" exitCode=0
Nov 02 14:34:28.862402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.861933    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66" event={"ID":"8c86bbed-1a24-46a1-9ece-8d624c9404e1","Type":"ContainerDied","Data":"9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176"}
Nov 02 14:34:28.862402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.861958    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"
Nov 02 14:34:28.862402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.861999    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66" event={"ID":"8c86bbed-1a24-46a1-9ece-8d624c9404e1","Type":"ContainerDied","Data":"11f2e3b40d665a2d734816e4d57b276a35a94fb8dd45dc7e5fc35bc41335d4ec"}
Nov 02 14:34:28.862402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.862025    5034 scope.go:122] "RemoveContainer" containerID="9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176"
Nov 02 14:34:28.875438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.875417    5034 scope.go:122] "RemoveContainer" containerID="9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176"
Nov 02 14:34:28.875736 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:34:28.875711    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176\": container with ID starting with 9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176 not found: ID does not exist" containerID="9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176"
Nov 02 14:34:28.875863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.875742    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176"} err="failed to get container status \"9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176\": rpc error: code = NotFound desc = could not find container \"9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176\": container with ID starting with 9a6b6998739b079c1da686d1ca1bb49d0e621c352d76254e3b71a4868a660176 not found: ID does not exist"
Nov 02 14:34:28.922038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.922007    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"]
Nov 02 14:34:28.943391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:28.943351    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5698cdb5c5-9nb66"]
Nov 02 14:34:29.073694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.073664    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:29.176934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.176850    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-kube-api-access-8ljhx\" (UniqueName: \"kubernetes.io/projected/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-kube-api-access-8ljhx\") pod \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") "
Nov 02 14:34:29.177122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.177002    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-openshift-ca\") pod \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") "
Nov 02 14:34:29.177242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.177217    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-xks-ca\") pod \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\" (UID: \"b93c95ca-53f4-4076-98b9-c156cb0d7d9b\") "
Nov 02 14:34:29.177563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.177532    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-openshift-ca" pod "b93c95ca-53f4-4076-98b9-c156cb0d7d9b" (UID: "b93c95ca-53f4-4076-98b9-c156cb0d7d9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:34:29.178022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.177995    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:29.179633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.179610    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-kube-api-access-8ljhx" pod "b93c95ca-53f4-4076-98b9-c156cb0d7d9b" (UID: "b93c95ca-53f4-4076-98b9-c156cb0d7d9b"). InnerVolumeSpecName "kube-api-access-8ljhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:34:29.180191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.180174    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-xks-ca" pod "b93c95ca-53f4-4076-98b9-c156cb0d7d9b" (UID: "b93c95ca-53f4-4076-98b9-c156cb0d7d9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:34:29.279463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.279436    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:29.279463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.279464    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljhx\" (UniqueName: \"kubernetes.io/projected/b93c95ca-53f4-4076-98b9-c156cb0d7d9b-kube-api-access-8ljhx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:34:29.399397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.399371    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c86bbed-1a24-46a1-9ece-8d624c9404e1" path="/var/lib/kubelet/pods/8c86bbed-1a24-46a1-9ece-8d624c9404e1/volumes"
Nov 02 14:34:29.875854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.875810    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p" event={"ID":"b93c95ca-53f4-4076-98b9-c156cb0d7d9b","Type":"ContainerDied","Data":"26a224581bef7221fd97bc92cee2da046d82c4f62ec66b952185d06451ea61ca"}
Nov 02 14:34:29.876305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.875840    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"
Nov 02 14:34:29.876860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:29.875878    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a224581bef7221fd97bc92cee2da046d82c4f62ec66b952185d06451ea61ca"
Nov 02 14:34:31.801092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.801055    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"]
Nov 02 14:34:31.801971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.801952    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8c86bbed-1a24-46a1-9ece-8d624c9404e1" containerName="payload-processing"
Nov 02 14:34:31.801971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.801968    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c86bbed-1a24-46a1-9ece-8d624c9404e1" containerName="payload-processing"
Nov 02 14:34:31.802162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.802007    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b93c95ca-53f4-4076-98b9-c156cb0d7d9b" containerName="revoke-keys"
Nov 02 14:34:31.802162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.802012    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b93c95ca-53f4-4076-98b9-c156cb0d7d9b" containerName="revoke-keys"
Nov 02 14:34:31.802162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.802128    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b93c95ca-53f4-4076-98b9-c156cb0d7d9b" containerName="revoke-keys"
Nov 02 14:34:31.802162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.802143    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c86bbed-1a24-46a1-9ece-8d624c9404e1" containerName="payload-processing"
Nov 02 14:34:31.806211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.806183    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:31.825648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.825619    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"]
Nov 02 14:34:31.831982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.831960    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-09d1a5\""
Nov 02 14:34:31.862272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.862244    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"]
Nov 02 14:34:31.867137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.867116    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:31.886884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.886846    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-09d1a5-dockercfg-r5p4n\""
Nov 02 14:34:31.895820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.895794    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"]
Nov 02 14:34:31.942819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.941583    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027cafe7-543b-49c1-80a8-8059071fd0b8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p\" (UID: \"027cafe7-543b-49c1-80a8-8059071fd0b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:31.942819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:31.941735    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tx2t\" (UniqueName: \"kubernetes.io/projected/027cafe7-543b-49c1-80a8-8059071fd0b8-kube-api-access-2tx2t\") pod \"payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p\" (UID: \"027cafe7-543b-49c1-80a8-8059071fd0b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:32.049563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.049524    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead84791-daf7-42de-8b50-91cd4bab519d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq\" (UID: \"ead84791-daf7-42de-8b50-91cd4bab519d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:32.049935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.049909    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tx2t\" (UniqueName: \"kubernetes.io/projected/027cafe7-543b-49c1-80a8-8059071fd0b8-kube-api-access-2tx2t\") pod \"payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p\" (UID: \"027cafe7-543b-49c1-80a8-8059071fd0b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:32.050105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.050088    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvzqk\" (UniqueName: \"kubernetes.io/projected/ead84791-daf7-42de-8b50-91cd4bab519d-kube-api-access-bvzqk\") pod \"payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq\" (UID: \"ead84791-daf7-42de-8b50-91cd4bab519d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:32.069846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.050515    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027cafe7-543b-49c1-80a8-8059071fd0b8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p\" (UID: \"027cafe7-543b-49c1-80a8-8059071fd0b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:32.089798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.071073    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027cafe7-543b-49c1-80a8-8059071fd0b8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p\" (UID: \"027cafe7-543b-49c1-80a8-8059071fd0b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:32.105477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.105446    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tx2t\" (UniqueName: \"kubernetes.io/projected/027cafe7-543b-49c1-80a8-8059071fd0b8-kube-api-access-2tx2t\") pod \"payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p\" (UID: \"027cafe7-543b-49c1-80a8-8059071fd0b8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:32.151943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.151911    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:32.221614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.221575    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead84791-daf7-42de-8b50-91cd4bab519d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq\" (UID: \"ead84791-daf7-42de-8b50-91cd4bab519d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:32.228849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.227979    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvzqk\" (UniqueName: \"kubernetes.io/projected/ead84791-daf7-42de-8b50-91cd4bab519d-kube-api-access-bvzqk\") pod \"payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq\" (UID: \"ead84791-daf7-42de-8b50-91cd4bab519d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:32.228849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.222809    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead84791-daf7-42de-8b50-91cd4bab519d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq\" (UID: \"ead84791-daf7-42de-8b50-91cd4bab519d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:32.254940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.254878    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvzqk\" (UniqueName: \"kubernetes.io/projected/ead84791-daf7-42de-8b50-91cd4bab519d-kube-api-access-bvzqk\") pod \"payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq\" (UID: \"ead84791-daf7-42de-8b50-91cd4bab519d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:32.491715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.491681    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:32.542008 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:34:32.541968    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027cafe7_543b_49c1_80a8_8059071fd0b8.slice/crio-2ac9266e3eaa5001ddbb0ce1bffb8738d3e208fc2ffabb9d0d1ef2c118fdad5e WatchSource:0}: Error finding container 2ac9266e3eaa5001ddbb0ce1bffb8738d3e208fc2ffabb9d0d1ef2c118fdad5e: Status 404 returned error can't find the container with id 2ac9266e3eaa5001ddbb0ce1bffb8738d3e208fc2ffabb9d0d1ef2c118fdad5e
Nov 02 14:34:32.544543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.544520    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"]
Nov 02 14:34:32.817829 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:34:32.817803    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead84791_daf7_42de_8b50_91cd4bab519d.slice/crio-edaf1be5a02233709442006c6ac9e9045436e3ab3eedc3116d21138f5a843471 WatchSource:0}: Error finding container edaf1be5a02233709442006c6ac9e9045436e3ab3eedc3116d21138f5a843471: Status 404 returned error can't find the container with id edaf1be5a02233709442006c6ac9e9045436e3ab3eedc3116d21138f5a843471
Nov 02 14:34:32.822544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.822521    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"]
Nov 02 14:34:32.922286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.922257    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p" event={"ID":"027cafe7-543b-49c1-80a8-8059071fd0b8","Type":"ContainerStarted","Data":"6a94cc7c4dc6115070e97ff1637ac75ded26775a44ef71598223d4035aa1a052"}
Nov 02 14:34:32.922463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.922321    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p" event={"ID":"027cafe7-543b-49c1-80a8-8059071fd0b8","Type":"ContainerStarted","Data":"2ac9266e3eaa5001ddbb0ce1bffb8738d3e208fc2ffabb9d0d1ef2c118fdad5e"}
Nov 02 14:34:32.922539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.922498    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:32.924410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.924388    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq" event={"ID":"ead84791-daf7-42de-8b50-91cd4bab519d","Type":"ContainerStarted","Data":"397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33"}
Nov 02 14:34:32.924518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.924432    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq" event={"ID":"ead84791-daf7-42de-8b50-91cd4bab519d","Type":"ContainerStarted","Data":"edaf1be5a02233709442006c6ac9e9045436e3ab3eedc3116d21138f5a843471"}
Nov 02 14:34:32.924518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.924464    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:32.969706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:32.969667    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p" podStartSLOduration=1.9696547679999998 podStartE2EDuration="1.969654768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:34:32.965368775 +0000 UTC m=+3063.716673213" watchObservedRunningTime="2026-11-02 14:34:32.969654768 +0000 UTC m=+3063.720959206"
Nov 02 14:34:33.011520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:33.011471    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq" podStartSLOduration=2.011455604 podStartE2EDuration="2.011455604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:34:33.008072418 +0000 UTC m=+3063.759376858" watchObservedRunningTime="2026-11-02 14:34:33.011455604 +0000 UTC m=+3063.762760043"
Nov 02 14:34:36.034558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:36.034522    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"]
Nov 02 14:34:36.047873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:36.047838    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-960ec365-4zzlx"]
Nov 02 14:34:37.399346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:37.399311    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c56a91-d36f-4620-a00e-d3fc5948ec15" path="/var/lib/kubelet/pods/68c56a91-d36f-4620-a00e-d3fc5948ec15/volumes"
Nov 02 14:34:37.556043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:37.556007    5034 scope.go:122] "RemoveContainer" containerID="3e4387897a21534bd4a5c53ee4ba9a5141f42b3bf14e278d519d0ada97ab1d13"
Nov 02 14:34:37.571148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:37.571113    5034 scope.go:122] "RemoveContainer" containerID="c0dedafc2900c2e4ea8121f96f0d8f579ce5c21eae5d15e11a150ffefb20825e"
Nov 02 14:34:37.583214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:37.583172    5034 scope.go:122] "RemoveContainer" containerID="ee105bae23b96aa18b54a22abfc34a99e3d03bf6d22f72a3006a10bfdb81c481"
Nov 02 14:34:43.937301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:43.937269    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:34:43.937951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:43.937350    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:34:48.417158 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:34:48.417117    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:34:49.032651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:49.032616    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"]
Nov 02 14:34:49.045764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:49.045731    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f05c5d36-462j8"]
Nov 02 14:34:49.458194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:49.458112    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8024c587-a435-4718-89d3-c461d97aeb18" path="/var/lib/kubelet/pods/8024c587-a435-4718-89d3-c461d97aeb18/volumes"
Nov 02 14:34:59.668158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.668127    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"]
Nov 02 14:34:59.676841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.676587    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.693969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.693845    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"]
Nov 02 14:34:59.697698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.695172    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"agc-cred-5b4b8e78c292\""
Nov 02 14:34:59.697698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.695228    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"kube-root-ca.crt\""
Nov 02 14:34:59.708934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.708912    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"payload-processing-external-model-e2e--47c96fd3-dockercfg-cfmjx\""
Nov 02 14:34:59.709144 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.708951    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-66bc6d\""
Nov 02 14:34:59.709286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.708992    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"openshift-service-ca.crt\""
Nov 02 14:34:59.709392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.709052    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-66bc6d\"/\"routing-overlay\""
Nov 02 14:34:59.787689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.787648    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.788043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.788017    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.788283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.788264    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t82q\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-kube-api-access-6t82q\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.788505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.788483    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.893146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.893086    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.894031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.894002    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.898889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.898864    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t82q\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-kube-api-access-6t82q\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.908318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.908298    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.917104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.894926    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.917277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.910174    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.945842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.932711    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t82q\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-kube-api-access-6t82q\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:34:59.978426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:34:59.978389    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") " pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:35:00.027800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:00.027752    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:35:00.777657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:00.777606    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"]
Nov 02 14:35:01.647877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:01.632457    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm" event={"ID":"0317cec9-bba9-40bb-8efe-a6f979baae77","Type":"ContainerStarted","Data":"324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6"}
Nov 02 14:35:01.647877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:01.632539    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm" event={"ID":"0317cec9-bba9-40bb-8efe-a6f979baae77","Type":"ContainerStarted","Data":"6b906660b10066bc2d80cf9a80c05cc1abea08507a80765b4557e09d51355415"}
Nov 02 14:35:01.647877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:01.633915    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:35:01.741804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:01.738801    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm" podStartSLOduration=2.738763144 podStartE2EDuration="2.738763144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:01.704360079 +0000 UTC m=+3092.455664511" watchObservedRunningTime="2026-11-02 14:35:01.738763144 +0000 UTC m=+3092.490067586"
Nov 02 14:35:04.035166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:04.035123    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"]
Nov 02 14:35:04.047208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:04.047177    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa18ab9a-d24t9"]
Nov 02 14:35:05.460802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:05.454815    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04c74a77-d231-4dc7-a08a-04110d047bd9" path="/var/lib/kubelet/pods/04c74a77-d231-4dc7-a08a-04110d047bd9/volumes"
Nov 02 14:35:08.237231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:08.237200    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"]
Nov 02 14:35:08.241145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:08.241119    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:35:13.237725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.237624    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm" podUID="0317cec9-bba9-40bb-8efe-a6f979baae77" containerName="payload-processing" containerID="cri-o://324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6" gracePeriod=25
Nov 02 14:35:13.629441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.629415    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:35:13.667527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.667490    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"]
Nov 02 14:35:13.668915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.668893    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0317cec9-bba9-40bb-8efe-a6f979baae77" containerName="payload-processing"
Nov 02 14:35:13.669041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.668916    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0317cec9-bba9-40bb-8efe-a6f979baae77" containerName="payload-processing"
Nov 02 14:35:13.669236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.669222    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0317cec9-bba9-40bb-8efe-a6f979baae77" containerName="payload-processing"
Nov 02 14:35:13.674139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.674116    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.699117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.699084    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e1fcb4-openshift-default-dockercfg-jk4wn\""
Nov 02 14:35:13.701493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.701468    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-plugins-config-volume\") pod \"0317cec9-bba9-40bb-8efe-a6f979baae77\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") "
Nov 02 14:35:13.701668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.701652    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-kube-api-access-6t82q\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-kube-api-access-6t82q\") pod \"0317cec9-bba9-40bb-8efe-a6f979baae77\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") "
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.701891    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-routing-overlay-volume\") pod \"0317cec9-bba9-40bb-8efe-a6f979baae77\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") "
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.701953    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-provider-credentials\") pod \"0317cec9-bba9-40bb-8efe-a6f979baae77\" (UID: \"0317cec9-bba9-40bb-8efe-a6f979baae77\") "
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.702241    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65501c91-8e19-4ee7-bb4c-ec746643d814-istiod-ca-cert\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.702290    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-data\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.702314    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-token\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.702358    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-socket\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.702543    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-podinfo\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.702652    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-envoy\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.702818    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-certs\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.702919    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-credential-socket\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.702947    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79x8\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-kube-api-access-d79x8\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.703406    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"]
Nov 02 14:35:13.703556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.703550    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-plugins-config-volume" pod "0317cec9-bba9-40bb-8efe-a6f979baae77" (UID: "0317cec9-bba9-40bb-8efe-a6f979baae77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:35:13.704443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.704392    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-routing-overlay-volume" pod "0317cec9-bba9-40bb-8efe-a6f979baae77" (UID: "0317cec9-bba9-40bb-8efe-a6f979baae77"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:35:13.713995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.713972    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-kube-api-access-6t82q" pod "0317cec9-bba9-40bb-8efe-a6f979baae77" (UID: "0317cec9-bba9-40bb-8efe-a6f979baae77"). InnerVolumeSpecName "kube-api-access-6t82q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:35:13.715057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.715039    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-provider-credentials" pod "0317cec9-bba9-40bb-8efe-a6f979baae77" (UID: "0317cec9-bba9-40bb-8efe-a6f979baae77"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:35:13.881858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.844065    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-certs\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:13.924802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.914947    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-credential-socket\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.002914    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79x8\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-kube-api-access-d79x8\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003018    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65501c91-8e19-4ee7-bb4c-ec746643d814-istiod-ca-cert\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003076    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-data\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003102    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-token\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003144    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-socket\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003345    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-podinfo\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003462    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-envoy\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003679    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t82q\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-kube-api-access-6t82q\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003703    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003720    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0317cec9-bba9-40bb-8efe-a6f979baae77-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.003735    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0317cec9-bba9-40bb-8efe-a6f979baae77-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.005653    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65501c91-8e19-4ee7-bb4c-ec746643d814-istiod-ca-cert\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.925393    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-credential-socket\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.006800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:13.864122    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-certs\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.007698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.007271    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-socket\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.007698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.007552    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-data\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.042481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.035815    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-podinfo\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.057039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.047516    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-envoy\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.057039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.048114    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-token\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.057039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.056415    5034 generic.go:356] "Generic (PLEG): container finished" podID="0317cec9-bba9-40bb-8efe-a6f979baae77" containerID="324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6" exitCode=0
Nov 02 14:35:14.057039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.056565    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79x8\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-kube-api-access-d79x8\") pod \"e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") " pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.057039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.056671    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"
Nov 02 14:35:14.057039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.056813    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm" event={"ID":"0317cec9-bba9-40bb-8efe-a6f979baae77","Type":"ContainerDied","Data":"324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6"}
Nov 02 14:35:14.057039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.056856    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm" event={"ID":"0317cec9-bba9-40bb-8efe-a6f979baae77","Type":"ContainerDied","Data":"6b906660b10066bc2d80cf9a80c05cc1abea08507a80765b4557e09d51355415"}
Nov 02 14:35:14.057039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.056881    5034 scope.go:122] "RemoveContainer" containerID="324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6"
Nov 02 14:35:14.078035 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.078014    5034 scope.go:122] "RemoveContainer" containerID="324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6"
Nov 02 14:35:14.078358 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:35:14.078333    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6\": container with ID starting with 324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6 not found: ID does not exist" containerID="324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6"
Nov 02 14:35:14.078423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.078366    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6"} err="failed to get container status \"324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6\": rpc error: code = NotFound desc = could not find container \"324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6\": container with ID starting with 324a2fbef346d00ad15fac144aad0add209b49f0658d40f8b481c3b1d2e428a6 not found: ID does not exist"
Nov 02 14:35:14.124199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.124172    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"]
Nov 02 14:35:14.131142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.131115    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-66bc6d/payload-processing-external-model-e2e-extmod-b-66bc6d-7447k2czm"]
Nov 02 14:35:14.353466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.353433    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:14.686193 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.686167    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"]
Nov 02 14:35:14.688089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.688053    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:35:14.688190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.688122    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:35:14.688190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:14.688151    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:35:15.070621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:15.070582    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" event={"ID":"65501c91-8e19-4ee7-bb4c-ec746643d814","Type":"ContainerStarted","Data":"1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e"}
Nov 02 14:35:15.070890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:15.070673    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" event={"ID":"65501c91-8e19-4ee7-bb4c-ec746643d814","Type":"ContainerStarted","Data":"6ef555936a1f15048ce4a8262f8b766282d7bf577c347670339adc9ecaa1d75a"}
Nov 02 14:35:15.158769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:15.158704    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" podStartSLOduration=2.158683934 podStartE2EDuration="2.158683934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:15.149628062 +0000 UTC m=+3105.900932501" watchObservedRunningTime="2026-11-02 14:35:15.158683934 +0000 UTC m=+3105.909988374"
Nov 02 14:35:15.357933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:15.357839    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:15.367512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:15.366858    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:15.403050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:15.403005    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0317cec9-bba9-40bb-8efe-a6f979baae77" path="/var/lib/kubelet/pods/0317cec9-bba9-40bb-8efe-a6f979baae77/volumes"
Nov 02 14:35:16.045261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:16.045232    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"]
Nov 02 14:35:16.052532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:16.052506    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-394ada8c-zdjgn"]
Nov 02 14:35:16.081087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:16.081026    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:16.082198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:16.082171    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:35:17.421518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:17.421485    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f451c332-f287-45c4-8286-9844268d9bd6" path="/var/lib/kubelet/pods/f451c332-f287-45c4-8286-9844268d9bd6/volumes"
Nov 02 14:35:18.493035 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:35:18.492962    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:35:20.581808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.568175    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"]
Nov 02 14:35:20.581808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.575865    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:20.593806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.584664    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"]
Nov 02 14:35:20.620223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.620139    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e1fcb4\""
Nov 02 14:35:20.620430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.620270    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e1fcb4-dockercfg-c9jt8\""
Nov 02 14:35:20.694601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.688629    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de00d3b1-f2d5-46c7-a29c-fb1576200196-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz\" (UID: \"de00d3b1-f2d5-46c7-a29c-fb1576200196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:20.694601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.688911    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j4xf\" (UniqueName: \"kubernetes.io/projected/de00d3b1-f2d5-46c7-a29c-fb1576200196-kube-api-access-7j4xf\") pod \"payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz\" (UID: \"de00d3b1-f2d5-46c7-a29c-fb1576200196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:20.791111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.791074    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de00d3b1-f2d5-46c7-a29c-fb1576200196-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz\" (UID: \"de00d3b1-f2d5-46c7-a29c-fb1576200196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:20.791324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.791218    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j4xf\" (UniqueName: \"kubernetes.io/projected/de00d3b1-f2d5-46c7-a29c-fb1576200196-kube-api-access-7j4xf\") pod \"payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz\" (UID: \"de00d3b1-f2d5-46c7-a29c-fb1576200196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:20.791818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.791754    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de00d3b1-f2d5-46c7-a29c-fb1576200196-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz\" (UID: \"de00d3b1-f2d5-46c7-a29c-fb1576200196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:20.816870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:20.816834    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j4xf\" (UniqueName: \"kubernetes.io/projected/de00d3b1-f2d5-46c7-a29c-fb1576200196-kube-api-access-7j4xf\") pod \"payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz\" (UID: \"de00d3b1-f2d5-46c7-a29c-fb1576200196\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:21.001312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:21.001280    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:21.319401 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:35:21.319365    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde00d3b1_f2d5_46c7_a29c_fb1576200196.slice/crio-2e4bb5e8dc3c72c2f89047e4b7b067c5ac26930b588eb728a23b8cd2c55fc7c8 WatchSource:0}: Error finding container 2e4bb5e8dc3c72c2f89047e4b7b067c5ac26930b588eb728a23b8cd2c55fc7c8: Status 404 returned error can't find the container with id 2e4bb5e8dc3c72c2f89047e4b7b067c5ac26930b588eb728a23b8cd2c55fc7c8
Nov 02 14:35:21.323605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:21.323577    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"]
Nov 02 14:35:22.031461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:22.031426    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"]
Nov 02 14:35:22.044466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:22.044400    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b91daf-shmbb"]
Nov 02 14:35:22.220340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:22.220301    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz" event={"ID":"de00d3b1-f2d5-46c7-a29c-fb1576200196","Type":"ContainerStarted","Data":"fd59f174a079fc1377c4ad159c94a7acb9899cea2e88cc5ddf51fa84d4b6b6f2"}
Nov 02 14:35:22.220546 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:22.220376    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz" event={"ID":"de00d3b1-f2d5-46c7-a29c-fb1576200196","Type":"ContainerStarted","Data":"2e4bb5e8dc3c72c2f89047e4b7b067c5ac26930b588eb728a23b8cd2c55fc7c8"}
Nov 02 14:35:22.220546 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:22.220434    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:22.265597 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:22.265548    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz" podStartSLOduration=2.265534714 podStartE2EDuration="2.265534714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:22.262922162 +0000 UTC m=+3113.014226590" watchObservedRunningTime="2026-11-02 14:35:22.265534714 +0000 UTC m=+3113.016839196"
Nov 02 14:35:23.400383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:23.400346    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d14778a4-3528-4a2c-b61c-f7ae977edeab" path="/var/lib/kubelet/pods/d14778a4-3528-4a2c-b61c-f7ae977edeab/volumes"
Nov 02 14:35:28.087532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:28.087487    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"]
Nov 02 14:35:28.106614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:28.106576    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4c0002-wmrl2"]
Nov 02 14:35:29.398268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:29.398235    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904a4b88-327f-4bfc-bf1e-28d0ad054106" path="/var/lib/kubelet/pods/904a4b88-327f-4bfc-bf1e-28d0ad054106/volumes"
Nov 02 14:35:33.233004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:33.232973    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:35:37.649424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:37.649386    5034 scope.go:122] "RemoveContainer" containerID="188ab232c006cdc461a77e7ba3bbfdadbd186db9eb2a9e98265f00566188570e"
Nov 02 14:35:37.661278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:37.661253    5034 scope.go:122] "RemoveContainer" containerID="89d9db0b0df685b258dadb2fb70811a4c93c46818cc0a74980768f369712d252"
Nov 02 14:35:37.672484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:37.672466    5034 scope.go:122] "RemoveContainer" containerID="43da236de8d1880690edd79e36b09fd6a11e183928227c7ab715690dbdb63bf4"
Nov 02 14:35:39.036961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:39.036922    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"]
Nov 02 14:35:39.049565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:39.049534    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"]
Nov 02 14:35:39.054227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:39.054201    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e0b5385-72jrx"]
Nov 02 14:35:39.058268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:39.058246    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24ef0910-j489g"]
Nov 02 14:35:39.397688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:39.397613    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d6b85e-f4cf-4ec3-aab9-d43d532c42f3" path="/var/lib/kubelet/pods/46d6b85e-f4cf-4ec3-aab9-d43d532c42f3/volumes"
Nov 02 14:35:39.398328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:39.398305    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b6bd36-75d9-4a4e-a569-f1d181b84c13" path="/var/lib/kubelet/pods/71b6bd36-75d9-4a4e-a569-f1d181b84c13/volumes"
Nov 02 14:35:43.312323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.312286    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"]
Nov 02 14:35:43.320414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.320390    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.336587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.336560    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"]
Nov 02 14:35:43.343573 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.343547    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"kube-root-ca.crt\""
Nov 02 14:35:43.343720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.343627    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a02c35\""
Nov 02 14:35:43.343720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.343717    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"routing-overlay\""
Nov 02 14:35:43.343893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.343801    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"agc-cred-8d633d498566\""
Nov 02 14:35:43.343893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.343836    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"agc-cred-b0311247be97\""
Nov 02 14:35:43.350355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.350107    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"openshift-service-ca.crt\""
Nov 02 14:35:43.350355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.350342    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a02c35\"/\"payload-processing-external-model-e2e--ac2d6d70-dockercfg-sd6gb\""
Nov 02 14:35:43.394592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.394561    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.394763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.394744    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.394865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.394848    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phxmx\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-kube-api-access-phxmx\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.394934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.394882    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.496180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.496148    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.496365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.496245    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phxmx\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-kube-api-access-phxmx\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.496365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.496278    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.496519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.496503    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.497124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.497092    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.497292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.497273    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.498875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.498846    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.521202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.521176    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phxmx\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-kube-api-access-phxmx\") pod \"payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") " pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.639698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.639632    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:43.955037 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:35:43.954998    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2ec348_84ea_42cf_bd18_fa657ec9f07d.slice/crio-d768afd81c98a277e110b38f6e72541d0a84db098b385daaf441696ebb47577e WatchSource:0}: Error finding container d768afd81c98a277e110b38f6e72541d0a84db098b385daaf441696ebb47577e: Status 404 returned error can't find the container with id d768afd81c98a277e110b38f6e72541d0a84db098b385daaf441696ebb47577e
Nov 02 14:35:43.955647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:43.955624    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"]
Nov 02 14:35:44.578602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:44.578560    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9" event={"ID":"0c2ec348-84ea-42cf-bd18-fa657ec9f07d","Type":"ContainerStarted","Data":"057d1b55a924e27ba37de3ab55c7d2c840e7fee4b366a3e86e6bdc8ef5f66581"}
Nov 02 14:35:44.579058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:44.578638    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:44.579058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:44.578663    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9" event={"ID":"0c2ec348-84ea-42cf-bd18-fa657ec9f07d","Type":"ContainerStarted","Data":"d768afd81c98a277e110b38f6e72541d0a84db098b385daaf441696ebb47577e"}
Nov 02 14:35:44.623594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:44.623545    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9" podStartSLOduration=1.6235325440000001 podStartE2EDuration="1.623532544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:44.622532372 +0000 UTC m=+3135.373836810" watchObservedRunningTime="2026-11-02 14:35:44.623532544 +0000 UTC m=+3135.374836981"
Nov 02 14:35:48.424207 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:35:48.424164    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:35:48.938541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:48.938502    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"]
Nov 02 14:35:48.951137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:48.951107    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:48.975544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:48.972098    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:35:48.982443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:48.982015    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"]
Nov 02 14:35:48.985603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:48.985347    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:35:48.985603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:48.985460    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:35:48.985603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:48.985566    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:35:49.024429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.024390    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2gv9\" (UniqueName: \"kubernetes.io/projected/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-kube-api-access-w2gv9\") pod \"maas-api-revoke-sub-a02d9552-rtmbr\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.027087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.027056    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a02d9552-rtmbr\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.027367 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.027349    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a02d9552-rtmbr\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.129265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.129228    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2gv9\" (UniqueName: \"kubernetes.io/projected/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-kube-api-access-w2gv9\") pod \"maas-api-revoke-sub-a02d9552-rtmbr\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.129444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.129287    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a02d9552-rtmbr\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.129444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.129337    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a02d9552-rtmbr\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.130068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.130041    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a02d9552-rtmbr\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.131707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.131684    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a02d9552-rtmbr\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.154128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.154101    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2gv9\" (UniqueName: \"kubernetes.io/projected/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-kube-api-access-w2gv9\") pod \"maas-api-revoke-sub-a02d9552-rtmbr\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.338800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.338556    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:49.546754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.546700    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"]
Nov 02 14:35:49.550034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.549991    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:49.772835 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:35:49.772770    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7aed91_0172_4904_bcf1_2ec1dd897a6d.slice/crio-77cbe561d5bdce86bfa0090038f9ba533c67ae45d5a69a4e4615d56a67383ab3 WatchSource:0}: Error finding container 77cbe561d5bdce86bfa0090038f9ba533c67ae45d5a69a4e4615d56a67383ab3: Status 404 returned error can't find the container with id 77cbe561d5bdce86bfa0090038f9ba533c67ae45d5a69a4e4615d56a67383ab3
Nov 02 14:35:49.773478 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:49.773454    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"]
Nov 02 14:35:50.636235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:50.636135    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr" event={"ID":"1e7aed91-0172-4904-bcf1-2ec1dd897a6d","Type":"ContainerStarted","Data":"d3253f0cc1cbdb380064f88f536d591a08b7d2ad4665fac8bad54dbf6661f384"}
Nov 02 14:35:50.636235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:50.636209    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr" event={"ID":"1e7aed91-0172-4904-bcf1-2ec1dd897a6d","Type":"ContainerStarted","Data":"77cbe561d5bdce86bfa0090038f9ba533c67ae45d5a69a4e4615d56a67383ab3"}
Nov 02 14:35:50.683473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:50.683412    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr" podStartSLOduration=2.683398209 podStartE2EDuration="2.683398209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:35:50.680522462 +0000 UTC m=+3141.431826903" watchObservedRunningTime="2026-11-02 14:35:50.683398209 +0000 UTC m=+3141.434702673"
Nov 02 14:35:51.648560 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:51.648517    5034 generic.go:356] "Generic (PLEG): container finished" podID="1e7aed91-0172-4904-bcf1-2ec1dd897a6d" containerID="d3253f0cc1cbdb380064f88f536d591a08b7d2ad4665fac8bad54dbf6661f384" exitCode=0
Nov 02 14:35:51.649017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:51.648556    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr" event={"ID":"1e7aed91-0172-4904-bcf1-2ec1dd897a6d","Type":"ContainerDied","Data":"d3253f0cc1cbdb380064f88f536d591a08b7d2ad4665fac8bad54dbf6661f384"}
Nov 02 14:35:52.859359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:52.859332    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:52.901356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:52.901320    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-xks-ca\") pod \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") "
Nov 02 14:35:52.901662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:52.901643    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-openshift-ca\") pod \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") "
Nov 02 14:35:52.901905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:52.901853    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-kube-api-access-w2gv9\" (UniqueName: \"kubernetes.io/projected/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-kube-api-access-w2gv9\") pod \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\" (UID: \"1e7aed91-0172-4904-bcf1-2ec1dd897a6d\") "
Nov 02 14:35:52.902049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:52.902017    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-openshift-ca" pod "1e7aed91-0172-4904-bcf1-2ec1dd897a6d" (UID: "1e7aed91-0172-4904-bcf1-2ec1dd897a6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:35:52.903335 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:52.903284    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:52.905031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:52.905001    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-kube-api-access-w2gv9" pod "1e7aed91-0172-4904-bcf1-2ec1dd897a6d" (UID: "1e7aed91-0172-4904-bcf1-2ec1dd897a6d"). InnerVolumeSpecName "kube-api-access-w2gv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:35:52.909361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:52.909300    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-xks-ca" pod "1e7aed91-0172-4904-bcf1-2ec1dd897a6d" (UID: "1e7aed91-0172-4904-bcf1-2ec1dd897a6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:35:53.004844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:53.004808    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2gv9\" (UniqueName: \"kubernetes.io/projected/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-kube-api-access-w2gv9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:53.004844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:53.004843    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e7aed91-0172-4904-bcf1-2ec1dd897a6d-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:53.669382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:53.669291    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"
Nov 02 14:35:53.669382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:53.669314    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr" event={"ID":"1e7aed91-0172-4904-bcf1-2ec1dd897a6d","Type":"ContainerDied","Data":"77cbe561d5bdce86bfa0090038f9ba533c67ae45d5a69a4e4615d56a67383ab3"}
Nov 02 14:35:53.669382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:53.669371    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77cbe561d5bdce86bfa0090038f9ba533c67ae45d5a69a4e4615d56a67383ab3"
Nov 02 14:35:54.548647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.548607    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9" podUID="0c2ec348-84ea-42cf-bd18-fa657ec9f07d" containerName="payload-processing" containerID="cri-o://057d1b55a924e27ba37de3ab55c7d2c840e7fee4b366a3e86e6bdc8ef5f66581" gracePeriod=25
Nov 02 14:35:54.684145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.684115    5034 generic.go:356] "Generic (PLEG): container finished" podID="0c2ec348-84ea-42cf-bd18-fa657ec9f07d" containerID="057d1b55a924e27ba37de3ab55c7d2c840e7fee4b366a3e86e6bdc8ef5f66581" exitCode=0
Nov 02 14:35:54.684297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.684181    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9" event={"ID":"0c2ec348-84ea-42cf-bd18-fa657ec9f07d","Type":"ContainerDied","Data":"057d1b55a924e27ba37de3ab55c7d2c840e7fee4b366a3e86e6bdc8ef5f66581"}
Nov 02 14:35:54.859809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.859772    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:54.935398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.935363    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-kube-api-access-phxmx\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-kube-api-access-phxmx\") pod \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") "
Nov 02 14:35:54.935584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.935538    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-provider-credentials\") pod \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") "
Nov 02 14:35:54.935720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.935698    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-plugins-config-volume\") pod \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") "
Nov 02 14:35:54.935819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.935796    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-routing-overlay-volume\") pod \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\" (UID: \"0c2ec348-84ea-42cf-bd18-fa657ec9f07d\") "
Nov 02 14:35:54.937151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.936311    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-plugins-config-volume" pod "0c2ec348-84ea-42cf-bd18-fa657ec9f07d" (UID: "0c2ec348-84ea-42cf-bd18-fa657ec9f07d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:35:54.937151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.936375    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-routing-overlay-volume" pod "0c2ec348-84ea-42cf-bd18-fa657ec9f07d" (UID: "0c2ec348-84ea-42cf-bd18-fa657ec9f07d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:35:54.937151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.936573    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:54.937151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.936589    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:54.938407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.938380    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-kube-api-access-phxmx" pod "0c2ec348-84ea-42cf-bd18-fa657ec9f07d" (UID: "0c2ec348-84ea-42cf-bd18-fa657ec9f07d"). InnerVolumeSpecName "kube-api-access-phxmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:35:54.942944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:54.942916    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-provider-credentials" pod "0c2ec348-84ea-42cf-bd18-fa657ec9f07d" (UID: "0c2ec348-84ea-42cf-bd18-fa657ec9f07d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:35:55.052801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:55.045372    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phxmx\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-kube-api-access-phxmx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:55.052801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:55.045413    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2ec348-84ea-42cf-bd18-fa657ec9f07d-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:35:55.697550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:55.697464    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"
Nov 02 14:35:55.697550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:55.697500    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9" event={"ID":"0c2ec348-84ea-42cf-bd18-fa657ec9f07d","Type":"ContainerDied","Data":"d768afd81c98a277e110b38f6e72541d0a84db098b385daaf441696ebb47577e"}
Nov 02 14:35:55.698100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:55.697567    5034 scope.go:122] "RemoveContainer" containerID="057d1b55a924e27ba37de3ab55c7d2c840e7fee4b366a3e86e6bdc8ef5f66581"
Nov 02 14:35:55.757908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:55.757879    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"]
Nov 02 14:35:55.770922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:55.770896    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a02c35/payload-processing-external-model-e2e-extmod-b-a02c35-867cg5rv9"]
Nov 02 14:35:57.399104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:57.399071    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2ec348-84ea-42cf-bd18-fa657ec9f07d" path="/var/lib/kubelet/pods/0c2ec348-84ea-42cf-bd18-fa657ec9f07d/volumes"
Nov 02 14:35:59.825639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.825604    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"]
Nov 02 14:35:59.826545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.826528    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0c2ec348-84ea-42cf-bd18-fa657ec9f07d" containerName="payload-processing"
Nov 02 14:35:59.826620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.826545    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2ec348-84ea-42cf-bd18-fa657ec9f07d" containerName="payload-processing"
Nov 02 14:35:59.826620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.826559    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1e7aed91-0172-4904-bcf1-2ec1dd897a6d" containerName="revoke-keys"
Nov 02 14:35:59.826620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.826564    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7aed91-0172-4904-bcf1-2ec1dd897a6d" containerName="revoke-keys"
Nov 02 14:35:59.826729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.826697    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2ec348-84ea-42cf-bd18-fa657ec9f07d" containerName="payload-processing"
Nov 02 14:35:59.826729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.826707    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7aed91-0172-4904-bcf1-2ec1dd897a6d" containerName="revoke-keys"
Nov 02 14:35:59.832016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.831997    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:35:59.851536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.851504    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"]
Nov 02 14:35:59.858973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.858946    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:35:59.859108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.859000    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:35:59.859259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.859236    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:35:59.861859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.861842    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:35:59.917928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.917890    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8d8d359-6g6lm\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:35:59.918096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.918010    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpx2q\" (UniqueName: \"kubernetes.io/projected/d8028d5e-350d-4c29-8634-15422df32bb9-kube-api-access-zpx2q\") pod \"maas-api-revoke-sub-c8d8d359-6g6lm\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:35:59.918169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:35:59.918149    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8d8d359-6g6lm\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:00.019798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:00.019751    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8d8d359-6g6lm\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:00.019980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:00.019870    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpx2q\" (UniqueName: \"kubernetes.io/projected/d8028d5e-350d-4c29-8634-15422df32bb9-kube-api-access-zpx2q\") pod \"maas-api-revoke-sub-c8d8d359-6g6lm\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:00.019980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:00.019955    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8d8d359-6g6lm\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:00.020386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:00.020363    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8d8d359-6g6lm\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:00.022378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:00.022336    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8d8d359-6g6lm\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:00.046407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:00.046385    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpx2q\" (UniqueName: \"kubernetes.io/projected/d8028d5e-350d-4c29-8634-15422df32bb9-kube-api-access-zpx2q\") pod \"maas-api-revoke-sub-c8d8d359-6g6lm\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:00.147863 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:00.147701    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:00.470119 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:36:00.470088    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8028d5e_350d_4c29_8634_15422df32bb9.slice/crio-9d389cb1f62177546e401d05f31ffd4aa7e6400380269bd816cb6340bc09ef1b WatchSource:0}: Error finding container 9d389cb1f62177546e401d05f31ffd4aa7e6400380269bd816cb6340bc09ef1b: Status 404 returned error can't find the container with id 9d389cb1f62177546e401d05f31ffd4aa7e6400380269bd816cb6340bc09ef1b
Nov 02 14:36:00.473768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:00.473736    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"]
Nov 02 14:36:00.747404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:00.747370    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm" event={"ID":"d8028d5e-350d-4c29-8634-15422df32bb9","Type":"ContainerStarted","Data":"9d389cb1f62177546e401d05f31ffd4aa7e6400380269bd816cb6340bc09ef1b"}
Nov 02 14:36:01.758661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:01.758631    5034 generic.go:356] "Generic (PLEG): container finished" podID="d8028d5e-350d-4c29-8634-15422df32bb9" containerID="7338b7bf00921c1ed5f939cda82a59578778cc8e976efd9457d44bc31265fb3b" exitCode=0
Nov 02 14:36:01.759084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:01.758712    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm" event={"ID":"d8028d5e-350d-4c29-8634-15422df32bb9","Type":"ContainerDied","Data":"7338b7bf00921c1ed5f939cda82a59578778cc8e976efd9457d44bc31265fb3b"}
Nov 02 14:36:02.976281 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:02.976251    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:03.068048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.067965    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8028d5e-350d-4c29-8634-15422df32bb9-kube-api-access-zpx2q\" (UniqueName: \"kubernetes.io/projected/d8028d5e-350d-4c29-8634-15422df32bb9-kube-api-access-zpx2q\") pod \"d8028d5e-350d-4c29-8634-15422df32bb9\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") "
Nov 02 14:36:03.068048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.068025    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-openshift-ca\") pod \"d8028d5e-350d-4c29-8634-15422df32bb9\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") "
Nov 02 14:36:03.068283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.068089    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-xks-ca\") pod \"d8028d5e-350d-4c29-8634-15422df32bb9\" (UID: \"d8028d5e-350d-4c29-8634-15422df32bb9\") "
Nov 02 14:36:03.068420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.068395    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-openshift-ca" pod "d8028d5e-350d-4c29-8634-15422df32bb9" (UID: "d8028d5e-350d-4c29-8634-15422df32bb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:03.068844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.068819    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:36:03.070995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.070968    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8028d5e-350d-4c29-8634-15422df32bb9-kube-api-access-zpx2q" pod "d8028d5e-350d-4c29-8634-15422df32bb9" (UID: "d8028d5e-350d-4c29-8634-15422df32bb9"). InnerVolumeSpecName "kube-api-access-zpx2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:03.088029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.087268    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-xks-ca" pod "d8028d5e-350d-4c29-8634-15422df32bb9" (UID: "d8028d5e-350d-4c29-8634-15422df32bb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:03.171171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.171143    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpx2q\" (UniqueName: \"kubernetes.io/projected/d8028d5e-350d-4c29-8634-15422df32bb9-kube-api-access-zpx2q\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:36:03.171291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.171174    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8028d5e-350d-4c29-8634-15422df32bb9-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:36:03.779396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.779363    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"
Nov 02 14:36:03.779596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.779406    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm" event={"ID":"d8028d5e-350d-4c29-8634-15422df32bb9","Type":"ContainerDied","Data":"9d389cb1f62177546e401d05f31ffd4aa7e6400380269bd816cb6340bc09ef1b"}
Nov 02 14:36:03.779596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:03.779507    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d389cb1f62177546e401d05f31ffd4aa7e6400380269bd816cb6340bc09ef1b"
Nov 02 14:36:18.419368 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:36:18.419326    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:36:23.611800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.606177    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"]
Nov 02 14:36:23.611800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.607738    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d8028d5e-350d-4c29-8634-15422df32bb9" containerName="revoke-keys"
Nov 02 14:36:23.611800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.607760    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8028d5e-350d-4c29-8634-15422df32bb9" containerName="revoke-keys"
Nov 02 14:36:23.611800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.608026    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8028d5e-350d-4c29-8634-15422df32bb9" containerName="revoke-keys"
Nov 02 14:36:23.684032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.683994    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.779136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.779107    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fe1881-openshift-default-dockercfg-s8sps\""
Nov 02 14:36:23.782245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.782214    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"]
Nov 02 14:36:23.796959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.796926    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-data\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.797115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.796984    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-podinfo\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.797256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.797231    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-token\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.797333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.797276    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-credential-socket\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.797396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.797331    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istiod-ca-cert\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.797448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.797407    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-envoy\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.797506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.797485    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-socket\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.797570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.797522    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-certs\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.797628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.797588    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxc9\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-kube-api-access-tzxc9\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.898710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.898625    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-podinfo\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.898921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.898822    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-token\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.898921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.898857    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-credential-socket\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.898921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.898903    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istiod-ca-cert\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.899124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.898973    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-envoy\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.899124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.899055    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-socket\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.899124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.899093    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-certs\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.899293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.899160    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxc9\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-kube-api-access-tzxc9\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.899293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.899268    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-data\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.899463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.899427    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-socket\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.899570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.899545    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-certs\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.899748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.899677    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-data\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.899963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.899941    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-credential-socket\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.900267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.900241    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istiod-ca-cert\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.901519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.901491    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-podinfo\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.901605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.901574    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-envoy\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.929129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.929101    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-token\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:23.940318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:23.940298    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxc9\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-kube-api-access-tzxc9\") pod \"e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") " pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:24.100687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:24.100653    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:24.453398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:24.453320    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"]
Nov 02 14:36:24.457446 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:36:24.457418    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1bf78b_d6af_48b8_b6c7_44c8f4694240.slice/crio-350ddf995e46630c627dfd8d92ebe33f73ddb2cf8e4ecaa48b5e0533d614ea0e WatchSource:0}: Error finding container 350ddf995e46630c627dfd8d92ebe33f73ddb2cf8e4ecaa48b5e0533d614ea0e: Status 404 returned error can't find the container with id 350ddf995e46630c627dfd8d92ebe33f73ddb2cf8e4ecaa48b5e0533d614ea0e
Nov 02 14:36:24.460675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:24.460634    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:36:24.460757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:24.460715    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:36:24.460757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:24.460749    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:36:25.142958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:25.142918    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" event={"ID":"2f1bf78b-d6af-48b8-b6c7-44c8f4694240","Type":"ContainerStarted","Data":"116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee"}
Nov 02 14:36:25.143477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:25.142974    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" event={"ID":"2f1bf78b-d6af-48b8-b6c7-44c8f4694240","Type":"ContainerStarted","Data":"350ddf995e46630c627dfd8d92ebe33f73ddb2cf8e4ecaa48b5e0533d614ea0e"}
Nov 02 14:36:25.221005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:25.220937    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" podStartSLOduration=2.220918653 podStartE2EDuration="2.220918653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:25.218466349 +0000 UTC m=+3175.969770809" watchObservedRunningTime="2026-11-02 14:36:25.220918653 +0000 UTC m=+3175.972223092"
Nov 02 14:36:26.101868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:26.101821    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:26.109179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:26.109150    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:26.153366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:26.153340    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:26.154276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:26.154254    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:36:37.730306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:37.730276    5034 scope.go:122] "RemoveContainer" containerID="ee940b3a2a8303c875cfcef0822a96db4116e472b698e2c5e89f358420b3787a"
Nov 02 14:36:37.742701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:37.742683    5034 scope.go:122] "RemoveContainer" containerID="0db51c5a9c61b3ef323a84ec3aee022b96b4f37f60d62a9cd3dd1531a2a1fbf0"
Nov 02 14:36:37.753522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:37.753494    5034 scope.go:122] "RemoveContainer" containerID="b5649419fdebdd510bec8b13ddcd1deead5dbb0a521b7cfded397fe9393c1996"
Nov 02 14:36:37.764314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:37.764295    5034 scope.go:122] "RemoveContainer" containerID="e484945003a838940d4ffed7e6c386088a603420c71f8fd161e37e20849134ae"
Nov 02 14:36:37.775460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:37.775442    5034 scope.go:122] "RemoveContainer" containerID="24ad91f19c37de8838ffdfa05d97449bce54df3376b7547a23330985a13d3c1f"
Nov 02 14:36:37.788633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:37.788604    5034 scope.go:122] "RemoveContainer" containerID="d16306f483b8561e7fc25a35c82d9bbfe505a16b3d86621444d11e90f901fbc9"
Nov 02 14:36:37.798696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:37.798678    5034 scope.go:122] "RemoveContainer" containerID="d89e24d0fa1e31af34282e792825c5ea5bb76fd958bae7bdbbcfb8f5fc82eb47"
Nov 02 14:36:39.979457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:39.976616    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"]
Nov 02 14:36:39.987148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:39.986042    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:39.987971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:39.987948    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"]
Nov 02 14:36:40.019093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.019063    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fe1881-dockercfg-dst9l\""
Nov 02 14:36:40.019093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.019089    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fe1881\""
Nov 02 14:36:40.060689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.060644    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97\" (UID: \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:40.060897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.060815    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcwtg\" (UniqueName: \"kubernetes.io/projected/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-kube-api-access-tcwtg\") pod \"payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97\" (UID: \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:40.162675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.162640    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97\" (UID: \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:40.162881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.162756    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcwtg\" (UniqueName: \"kubernetes.io/projected/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-kube-api-access-tcwtg\") pod \"payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97\" (UID: \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:40.163499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.163472    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97\" (UID: \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:40.197442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.197407    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcwtg\" (UniqueName: \"kubernetes.io/projected/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-kube-api-access-tcwtg\") pod \"payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97\" (UID: \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:40.320058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.320030    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:40.642754 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:36:40.642727    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb6543b_ac5e_49cc_845e_35c0b1d783d3.slice/crio-9c839ba28a3b9ad8f5ab81351f8720231e1d7c1556f7635a0567c8d6c4635b54 WatchSource:0}: Error finding container 9c839ba28a3b9ad8f5ab81351f8720231e1d7c1556f7635a0567c8d6c4635b54: Status 404 returned error can't find the container with id 9c839ba28a3b9ad8f5ab81351f8720231e1d7c1556f7635a0567c8d6c4635b54
Nov 02 14:36:40.647170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:40.647140    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"]
Nov 02 14:36:41.303497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:41.303463    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97" event={"ID":"5fb6543b-ac5e-49cc-845e-35c0b1d783d3","Type":"ContainerStarted","Data":"31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046"}
Nov 02 14:36:41.303976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:41.303518    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97" event={"ID":"5fb6543b-ac5e-49cc-845e-35c0b1d783d3","Type":"ContainerStarted","Data":"9c839ba28a3b9ad8f5ab81351f8720231e1d7c1556f7635a0567c8d6c4635b54"}
Nov 02 14:36:41.303976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:41.303623    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:41.352927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:41.352883    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97" podStartSLOduration=2.352869787 podStartE2EDuration="2.352869787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:41.348489971 +0000 UTC m=+3192.099794410" watchObservedRunningTime="2026-11-02 14:36:41.352869787 +0000 UTC m=+3192.104174256"
Nov 02 14:36:46.929043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:46.929010    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"]
Nov 02 14:36:46.934591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:46.934570    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:46.966635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:46.966608    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:36:46.966791 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:46.966643    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:36:46.966791 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:46.966714    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:36:46.966932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:46.966905    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:36:47.003797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.001651    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"]
Nov 02 14:36:47.050301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.050266    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e51146-25jng\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.050583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.050513    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e51146-25jng\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.050682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.050628    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vssw5\" (UniqueName: \"kubernetes.io/projected/6b6f7698-a0ef-4015-84b4-67b7f32c9641-kube-api-access-vssw5\") pod \"maas-api-revoke-sub-c1e51146-25jng\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.154765 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.151819    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e51146-25jng\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.154765 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.152023    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e51146-25jng\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.154765 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.152119    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vssw5\" (UniqueName: \"kubernetes.io/projected/6b6f7698-a0ef-4015-84b4-67b7f32c9641-kube-api-access-vssw5\") pod \"maas-api-revoke-sub-c1e51146-25jng\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.154765 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.153260    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e51146-25jng\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.156689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.156664    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e51146-25jng\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.202009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.201925    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vssw5\" (UniqueName: \"kubernetes.io/projected/6b6f7698-a0ef-4015-84b4-67b7f32c9641-kube-api-access-vssw5\") pod \"maas-api-revoke-sub-c1e51146-25jng\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.251095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.251064    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:47.617412 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:36:47.617376    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6f7698_a0ef_4015_84b4_67b7f32c9641.slice/crio-db6663201bf33e8ec7f2f3b2df3b6ef3496db58c793dc9bcd101f64bf4878a45 WatchSource:0}: Error finding container db6663201bf33e8ec7f2f3b2df3b6ef3496db58c793dc9bcd101f64bf4878a45: Status 404 returned error can't find the container with id db6663201bf33e8ec7f2f3b2df3b6ef3496db58c793dc9bcd101f64bf4878a45
Nov 02 14:36:47.617938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:47.617915    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"]
Nov 02 14:36:48.432229 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:36:48.432195    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:36:48.462292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:48.462261    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng" event={"ID":"6b6f7698-a0ef-4015-84b4-67b7f32c9641","Type":"ContainerStarted","Data":"677f377d8da08749fdeef06fcd2874dbefeb5c69345dfff1da38922cb8f59867"}
Nov 02 14:36:48.462456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:48.462320    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng" event={"ID":"6b6f7698-a0ef-4015-84b4-67b7f32c9641","Type":"ContainerStarted","Data":"db6663201bf33e8ec7f2f3b2df3b6ef3496db58c793dc9bcd101f64bf4878a45"}
Nov 02 14:36:48.516319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:48.516257    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng" podStartSLOduration=2.516239292 podStartE2EDuration="2.516239292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:48.516024638 +0000 UTC m=+3199.267329077" watchObservedRunningTime="2026-11-02 14:36:48.516239292 +0000 UTC m=+3199.267543733"
Nov 02 14:36:49.499811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:49.498645    5034 generic.go:356] "Generic (PLEG): container finished" podID="6b6f7698-a0ef-4015-84b4-67b7f32c9641" containerID="677f377d8da08749fdeef06fcd2874dbefeb5c69345dfff1da38922cb8f59867" exitCode=0
Nov 02 14:36:49.499811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:49.498760    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng" event={"ID":"6b6f7698-a0ef-4015-84b4-67b7f32c9641","Type":"ContainerDied","Data":"677f377d8da08749fdeef06fcd2874dbefeb5c69345dfff1da38922cb8f59867"}
Nov 02 14:36:50.773096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.773024    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:50.847764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.847727    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6f7698-a0ef-4015-84b4-67b7f32c9641-kube-api-access-vssw5\" (UniqueName: \"kubernetes.io/projected/6b6f7698-a0ef-4015-84b4-67b7f32c9641-kube-api-access-vssw5\") pod \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") "
Nov 02 14:36:50.848091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.847883    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-xks-ca\") pod \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") "
Nov 02 14:36:50.848091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.847959    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-openshift-ca\") pod \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\" (UID: \"6b6f7698-a0ef-4015-84b4-67b7f32c9641\") "
Nov 02 14:36:50.848564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.848539    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-openshift-ca" pod "6b6f7698-a0ef-4015-84b4-67b7f32c9641" (UID: "6b6f7698-a0ef-4015-84b4-67b7f32c9641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:36:50.849308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.849280    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:36:50.850948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.850925    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6f7698-a0ef-4015-84b4-67b7f32c9641-kube-api-access-vssw5" pod "6b6f7698-a0ef-4015-84b4-67b7f32c9641" (UID: "6b6f7698-a0ef-4015-84b4-67b7f32c9641"). InnerVolumeSpecName "kube-api-access-vssw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:36:50.862803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.862763    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-xks-ca" pod "6b6f7698-a0ef-4015-84b4-67b7f32c9641" (UID: "6b6f7698-a0ef-4015-84b4-67b7f32c9641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:36:50.951502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.951466    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vssw5\" (UniqueName: \"kubernetes.io/projected/6b6f7698-a0ef-4015-84b4-67b7f32c9641-kube-api-access-vssw5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:36:50.951502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:50.951505    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b6f7698-a0ef-4015-84b4-67b7f32c9641-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:36:51.523721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:51.523691    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"
Nov 02 14:36:51.523960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:51.523688    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng" event={"ID":"6b6f7698-a0ef-4015-84b4-67b7f32c9641","Type":"ContainerDied","Data":"db6663201bf33e8ec7f2f3b2df3b6ef3496db58c793dc9bcd101f64bf4878a45"}
Nov 02 14:36:51.523960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:51.523854    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db6663201bf33e8ec7f2f3b2df3b6ef3496db58c793dc9bcd101f64bf4878a45"
Nov 02 14:36:52.337106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:52.331069    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:36:54.043050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:54.043016    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"]
Nov 02 14:36:54.053029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:54.053001    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90e6d965-fcnpp"]
Nov 02 14:36:55.399161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:55.399128    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cddd3528-d314-410d-a4f5-c092f1c42fa5" path="/var/lib/kubelet/pods/cddd3528-d314-410d-a4f5-c092f1c42fa5/volumes"
Nov 02 14:36:55.559189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:55.559158    5034 generic.go:356] "Generic (PLEG): container finished" podID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerID="1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e" exitCode=0
Nov 02 14:36:55.559361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:55.559229    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" event={"ID":"65501c91-8e19-4ee7-bb4c-ec746643d814","Type":"ContainerDied","Data":"1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e"}
Nov 02 14:36:55.559673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:55.559653    5034 scope.go:122] "RemoveContainer" containerID="1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e"
Nov 02 14:36:55.561527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:55.561494    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:36:55.561599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:55.561572    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:36:55.561637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:55.561608    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:36:56.602991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:56.602934    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" event={"ID":"65501c91-8e19-4ee7-bb4c-ec746643d814","Type":"ContainerStarted","Data":"3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574"}
Nov 02 14:36:57.354254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:57.354226    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:36:57.355402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:57.355372    5034 patch_prober.go:29] interesting pod/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Nov 02 14:36:57.355539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:57.355425    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Nov 02 14:36:58.018604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.018570    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"]
Nov 02 14:36:58.019585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.019566    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6b6f7698-a0ef-4015-84b4-67b7f32c9641" containerName="revoke-keys"
Nov 02 14:36:58.019639 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.019587    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6f7698-a0ef-4015-84b4-67b7f32c9641" containerName="revoke-keys"
Nov 02 14:36:58.019751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.019742    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6f7698-a0ef-4015-84b4-67b7f32c9641" containerName="revoke-keys"
Nov 02 14:36:58.022709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.022691    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.039226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.039197    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:36:58.039330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.039202    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:36:58.039330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.039245    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:36:58.039330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.039253    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:36:58.058791 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.058753    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"]
Nov 02 14:36:58.112662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.112269    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa0e23a7-f5qq7\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.112889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.112682    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa0e23a7-f5qq7\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.112889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.112854    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp4c5\" (UniqueName: \"kubernetes.io/projected/f36e69ff-4bb2-4dd2-9009-11decc773623-kube-api-access-jp4c5\") pod \"maas-api-revoke-sub-fa0e23a7-f5qq7\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.214869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.214830    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp4c5\" (UniqueName: \"kubernetes.io/projected/f36e69ff-4bb2-4dd2-9009-11decc773623-kube-api-access-jp4c5\") pod \"maas-api-revoke-sub-fa0e23a7-f5qq7\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.215070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.214899    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa0e23a7-f5qq7\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.215766 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.215744    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa0e23a7-f5qq7\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.216516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.216416    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa0e23a7-f5qq7\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.217586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.217571    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa0e23a7-f5qq7\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.248434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.248411    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp4c5\" (UniqueName: \"kubernetes.io/projected/f36e69ff-4bb2-4dd2-9009-11decc773623-kube-api-access-jp4c5\") pod \"maas-api-revoke-sub-fa0e23a7-f5qq7\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.337837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.337740    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:36:58.354233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.354208    5034 patch_prober.go:29] interesting pod/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Nov 02 14:36:58.354352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.354274    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Nov 02 14:36:58.707254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:58.707225    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"]
Nov 02 14:36:59.356800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:59.356706    5034 patch_prober.go:29] interesting pod/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Nov 02 14:36:59.356800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:59.356765    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Nov 02 14:36:59.635489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:59.635406    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7" event={"ID":"f36e69ff-4bb2-4dd2-9009-11decc773623","Type":"ContainerStarted","Data":"492c393493a34c6540a93a3ded939333ec50f146a130aafc7b5ed626ebc7db29"}
Nov 02 14:36:59.635746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:59.635720    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7" event={"ID":"f36e69ff-4bb2-4dd2-9009-11decc773623","Type":"ContainerStarted","Data":"e141c4a55c0d83e0a52d7b8931f427c2acb0dca4ac48cd40372fc5b8041a81d6"}
Nov 02 14:36:59.723187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:36:59.723113    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7" podStartSLOduration=2.723091836 podStartE2EDuration="2.723091836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:36:59.696360745 +0000 UTC m=+3210.447665183" watchObservedRunningTime="2026-11-02 14:36:59.723091836 +0000 UTC m=+3210.474396275"
Nov 02 14:37:00.354694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:00.354617    5034 patch_prober.go:29] interesting pod/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Nov 02 14:37:00.354694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:00.354679    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Nov 02 14:37:00.647058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:00.646974    5034 generic.go:356] "Generic (PLEG): container finished" podID="f36e69ff-4bb2-4dd2-9009-11decc773623" containerID="492c393493a34c6540a93a3ded939333ec50f146a130aafc7b5ed626ebc7db29" exitCode=0
Nov 02 14:37:00.647557 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:00.647100    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7" event={"ID":"f36e69ff-4bb2-4dd2-9009-11decc773623","Type":"ContainerDied","Data":"492c393493a34c6540a93a3ded939333ec50f146a130aafc7b5ed626ebc7db29"}
Nov 02 14:37:01.091801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.090898    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:37:01.310806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.309717    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"]
Nov 02 14:37:01.319248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.315894    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.323801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.323748    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"]
Nov 02 14:37:01.350535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.335385    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e1136e-openshift-default-dockercfg-hr8x9\""
Nov 02 14:37:01.354286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.353927    5034 patch_prober.go:29] interesting pod/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Nov 02 14:37:01.354286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.353978    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Nov 02 14:37:01.520801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.457034    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-data\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.520801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.457137    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sdmz\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-kube-api-access-5sdmz\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.520801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.457216    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-envoy\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.520801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.457248    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istiod-ca-cert\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.520801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.457294    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-credential-socket\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.520801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.457357    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-podinfo\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.520801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.457460    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-socket\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.520801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.457493    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-certs\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.520801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.457611    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-token\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.578117    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-data\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.578212    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sdmz\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-kube-api-access-5sdmz\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.578287    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-envoy\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.578315    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istiod-ca-cert\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.578362    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-credential-socket\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.578425    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-podinfo\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.578527    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-socket\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.578562    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-certs\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.578676    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-token\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.672046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.579527    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-data\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.674133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.673742    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-credential-socket\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.675668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.675642    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istiod-ca-cert\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.676139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.676118    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-socket\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.705094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.683103    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-certs\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.768806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.767401    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-podinfo\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.768806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.767621    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-envoy\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.812802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.794976    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-token\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.848277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.847757    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sdmz\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-kube-api-access-5sdmz\") pod \"e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") " pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:01.953849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:01.953464    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:02.211846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.211349    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:37:02.250713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.248182    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-openshift-ca\") pod \"f36e69ff-4bb2-4dd2-9009-11decc773623\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") "
Nov 02 14:37:02.250713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.248351    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36e69ff-4bb2-4dd2-9009-11decc773623-kube-api-access-jp4c5\" (UniqueName: \"kubernetes.io/projected/f36e69ff-4bb2-4dd2-9009-11decc773623-kube-api-access-jp4c5\") pod \"f36e69ff-4bb2-4dd2-9009-11decc773623\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") "
Nov 02 14:37:02.250713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.248430    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-xks-ca\") pod \"f36e69ff-4bb2-4dd2-9009-11decc773623\" (UID: \"f36e69ff-4bb2-4dd2-9009-11decc773623\") "
Nov 02 14:37:02.250713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.248605    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-openshift-ca" pod "f36e69ff-4bb2-4dd2-9009-11decc773623" (UID: "f36e69ff-4bb2-4dd2-9009-11decc773623"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:37:02.250713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.249101    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:02.251191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.251074    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36e69ff-4bb2-4dd2-9009-11decc773623-kube-api-access-jp4c5" pod "f36e69ff-4bb2-4dd2-9009-11decc773623" (UID: "f36e69ff-4bb2-4dd2-9009-11decc773623"). InnerVolumeSpecName "kube-api-access-jp4c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:37:02.251251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.251201    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-xks-ca" pod "f36e69ff-4bb2-4dd2-9009-11decc773623" (UID: "f36e69ff-4bb2-4dd2-9009-11decc773623"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:37:02.356929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.355405    5034 patch_prober.go:29] interesting pod/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused" start-of-body=
Nov 02 14:37:02.356929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.355457    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.91:15021/healthz/ready\": dial tcp 10.132.1.91:15021: connect: connection refused"
Nov 02 14:37:02.356929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.355934    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp4c5\" (UniqueName: \"kubernetes.io/projected/f36e69ff-4bb2-4dd2-9009-11decc773623-kube-api-access-jp4c5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:02.356929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.355963    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f36e69ff-4bb2-4dd2-9009-11decc773623-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:02.359866 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:37:02.358690    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee48d5e3_cdfb_45e9_96ce_e5d75ff37979.slice/crio-ce5a0ddb2421d04c8766170a2c8d167a6019ee9c9132b1f2223147fe470a1cff WatchSource:0}: Error finding container ce5a0ddb2421d04c8766170a2c8d167a6019ee9c9132b1f2223147fe470a1cff: Status 404 returned error can't find the container with id ce5a0ddb2421d04c8766170a2c8d167a6019ee9c9132b1f2223147fe470a1cff
Nov 02 14:37:02.359866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.358809    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"]
Nov 02 14:37:02.362403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.362374    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:37:02.362510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.362447    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:37:02.362510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.362494    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:37:02.868417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.868389    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"
Nov 02 14:37:02.868978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.868423    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7" event={"ID":"f36e69ff-4bb2-4dd2-9009-11decc773623","Type":"ContainerDied","Data":"e141c4a55c0d83e0a52d7b8931f427c2acb0dca4ac48cd40372fc5b8041a81d6"}
Nov 02 14:37:02.868978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.868474    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e141c4a55c0d83e0a52d7b8931f427c2acb0dca4ac48cd40372fc5b8041a81d6"
Nov 02 14:37:02.870724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.870692    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" event={"ID":"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979","Type":"ContainerStarted","Data":"74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379"}
Nov 02 14:37:02.870918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.870748    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" event={"ID":"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979","Type":"ContainerStarted","Data":"ce5a0ddb2421d04c8766170a2c8d167a6019ee9c9132b1f2223147fe470a1cff"}
Nov 02 14:37:02.954818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.954738    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" podStartSLOduration=1.954719577 podStartE2EDuration="1.954719577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:02.949920739 +0000 UTC m=+3213.701225178" watchObservedRunningTime="2026-11-02 14:37:02.954719577 +0000 UTC m=+3213.706024016"
Nov 02 14:37:02.955531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:02.955501    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:03.359406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:03.359375    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:37:03.360306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:03.360286    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:37:03.961299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:03.961269    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:04.889762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:04.889728    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:04.890829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:04.890773    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:37:05.036082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:05.036054    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"]
Nov 02 14:37:05.041111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:05.041080    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a2f60a-vfg9d"]
Nov 02 14:37:05.398233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:05.398198    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55b7f19-c693-4fa2-9c01-03230f0f5924" path="/var/lib/kubelet/pods/c55b7f19-c693-4fa2-9c01-03230f0f5924/volumes"
Nov 02 14:37:07.773644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.773601    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"]
Nov 02 14:37:07.775074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.775047    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f36e69ff-4bb2-4dd2-9009-11decc773623" containerName="revoke-keys"
Nov 02 14:37:07.775074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.775073    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36e69ff-4bb2-4dd2-9009-11decc773623" containerName="revoke-keys"
Nov 02 14:37:07.775352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.775338    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36e69ff-4bb2-4dd2-9009-11decc773623" containerName="revoke-keys"
Nov 02 14:37:07.782362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.782284    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"]
Nov 02 14:37:07.782581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.782382    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:07.812691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.812666    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e1136e\""
Nov 02 14:37:07.813965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.813939    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e1136e-dockercfg-strqg\""
Nov 02 14:37:07.875035 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.874989    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c842308-c4ab-4068-bd2f-e1106361cb16-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj\" (UID: \"1c842308-c4ab-4068-bd2f-e1106361cb16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:07.875192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.875042    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2xf\" (UniqueName: \"kubernetes.io/projected/1c842308-c4ab-4068-bd2f-e1106361cb16-kube-api-access-xf2xf\") pod \"payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj\" (UID: \"1c842308-c4ab-4068-bd2f-e1106361cb16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:07.977397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.977356    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c842308-c4ab-4068-bd2f-e1106361cb16-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj\" (UID: \"1c842308-c4ab-4068-bd2f-e1106361cb16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:07.977397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.977396    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2xf\" (UniqueName: \"kubernetes.io/projected/1c842308-c4ab-4068-bd2f-e1106361cb16-kube-api-access-xf2xf\") pod \"payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj\" (UID: \"1c842308-c4ab-4068-bd2f-e1106361cb16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:07.978128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:07.978106    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c842308-c4ab-4068-bd2f-e1106361cb16-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj\" (UID: \"1c842308-c4ab-4068-bd2f-e1106361cb16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:08.035852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:08.035770    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2xf\" (UniqueName: \"kubernetes.io/projected/1c842308-c4ab-4068-bd2f-e1106361cb16-kube-api-access-xf2xf\") pod \"payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj\" (UID: \"1c842308-c4ab-4068-bd2f-e1106361cb16\") " pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:08.103995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:08.103967    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:08.427427 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:37:08.427397    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c842308_c4ab_4068_bd2f_e1106361cb16.slice/crio-140cc9f99fa499cd0c28dc060a6dd5545eaf8b7de3c093858916260b122ea332 WatchSource:0}: Error finding container 140cc9f99fa499cd0c28dc060a6dd5545eaf8b7de3c093858916260b122ea332: Status 404 returned error can't find the container with id 140cc9f99fa499cd0c28dc060a6dd5545eaf8b7de3c093858916260b122ea332
Nov 02 14:37:08.429070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:08.429046    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"]
Nov 02 14:37:08.964823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:08.964741    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj" event={"ID":"1c842308-c4ab-4068-bd2f-e1106361cb16","Type":"ContainerStarted","Data":"d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447"}
Nov 02 14:37:08.965311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:08.964835    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj" event={"ID":"1c842308-c4ab-4068-bd2f-e1106361cb16","Type":"ContainerStarted","Data":"140cc9f99fa499cd0c28dc060a6dd5545eaf8b7de3c093858916260b122ea332"}
Nov 02 14:37:08.965311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:08.964865    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:09.013805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:09.013741    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj" podStartSLOduration=2.013726313 podStartE2EDuration="2.013726313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:09.008022923 +0000 UTC m=+3219.759327361" watchObservedRunningTime="2026-11-02 14:37:09.013726313 +0000 UTC m=+3219.765030752"
Nov 02 14:37:10.532438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.532396    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"]
Nov 02 14:37:10.537884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.537860    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:10.587831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.580137    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:37:10.587831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.580181    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:37:10.587831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.581485    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3845c16-0135-41b6-bc87-953357282374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ff1f80-cxm99\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:10.587831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.581657    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3845c16-0135-41b6-bc87-953357282374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ff1f80-cxm99\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:10.587831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.581719    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxwb6\" (UniqueName: \"kubernetes.io/projected/c3845c16-0135-41b6-bc87-953357282374-kube-api-access-wxwb6\") pod \"maas-api-revoke-sub-c0ff1f80-cxm99\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:10.761488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.599138    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:37:10.761488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.653083    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:37:10.761488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.693390    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3845c16-0135-41b6-bc87-953357282374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ff1f80-cxm99\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:10.761488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.693468    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxwb6\" (UniqueName: \"kubernetes.io/projected/c3845c16-0135-41b6-bc87-953357282374-kube-api-access-wxwb6\") pod \"maas-api-revoke-sub-c0ff1f80-cxm99\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:10.761488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.693771    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3845c16-0135-41b6-bc87-953357282374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ff1f80-cxm99\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:10.761488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.718219    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3845c16-0135-41b6-bc87-953357282374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ff1f80-cxm99\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:10.804764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.803740    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxwb6\" (UniqueName: \"kubernetes.io/projected/c3845c16-0135-41b6-bc87-953357282374-kube-api-access-wxwb6\") pod \"maas-api-revoke-sub-c0ff1f80-cxm99\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:10.877395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.877137    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"]
Nov 02 14:37:10.896090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:10.894987    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3845c16-0135-41b6-bc87-953357282374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ff1f80-cxm99\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:11.179854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:11.179019    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:11.762644 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:37:11.762603    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3845c16_0135_41b6_bc87_953357282374.slice/crio-85f8f04a5b492e02a596fd3957736d63adf0849fbee69f703d58fd8de9cf5a87 WatchSource:0}: Error finding container 85f8f04a5b492e02a596fd3957736d63adf0849fbee69f703d58fd8de9cf5a87: Status 404 returned error can't find the container with id 85f8f04a5b492e02a596fd3957736d63adf0849fbee69f703d58fd8de9cf5a87
Nov 02 14:37:11.767590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:11.767563    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"]
Nov 02 14:37:12.464411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:12.464330    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99" event={"ID":"c3845c16-0135-41b6-bc87-953357282374","Type":"ContainerStarted","Data":"8b171b23b10ccc5c770472289b371dd918b88d9d255147f1bec7fb81ab85f9be"}
Nov 02 14:37:12.464411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:12.464395    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99" event={"ID":"c3845c16-0135-41b6-bc87-953357282374","Type":"ContainerStarted","Data":"85f8f04a5b492e02a596fd3957736d63adf0849fbee69f703d58fd8de9cf5a87"}
Nov 02 14:37:12.510122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:12.510076    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99" podStartSLOduration=2.510061603 podStartE2EDuration="2.510061603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:12.50822802 +0000 UTC m=+3223.259532458" watchObservedRunningTime="2026-11-02 14:37:12.510061603 +0000 UTC m=+3223.261366041"
Nov 02 14:37:13.475041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:13.474994    5034 generic.go:356] "Generic (PLEG): container finished" podID="c3845c16-0135-41b6-bc87-953357282374" containerID="8b171b23b10ccc5c770472289b371dd918b88d9d255147f1bec7fb81ab85f9be" exitCode=0
Nov 02 14:37:13.475461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:13.475069    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99" event={"ID":"c3845c16-0135-41b6-bc87-953357282374","Type":"ContainerDied","Data":"8b171b23b10ccc5c770472289b371dd918b88d9d255147f1bec7fb81ab85f9be"}
Nov 02 14:37:14.700869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.700836    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:14.750660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.750585    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3845c16-0135-41b6-bc87-953357282374-kube-api-access-wxwb6\" (UniqueName: \"kubernetes.io/projected/c3845c16-0135-41b6-bc87-953357282374-kube-api-access-wxwb6\") pod \"c3845c16-0135-41b6-bc87-953357282374\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") "
Nov 02 14:37:14.751049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.751027    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3845c16-0135-41b6-bc87-953357282374-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3845c16-0135-41b6-bc87-953357282374-maas-api-xks-ca\") pod \"c3845c16-0135-41b6-bc87-953357282374\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") "
Nov 02 14:37:14.751155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.751088    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3845c16-0135-41b6-bc87-953357282374-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3845c16-0135-41b6-bc87-953357282374-maas-api-openshift-ca\") pod \"c3845c16-0135-41b6-bc87-953357282374\" (UID: \"c3845c16-0135-41b6-bc87-953357282374\") "
Nov 02 14:37:14.751470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.751443    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3845c16-0135-41b6-bc87-953357282374-maas-api-openshift-ca" pod "c3845c16-0135-41b6-bc87-953357282374" (UID: "c3845c16-0135-41b6-bc87-953357282374"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:37:14.751943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.751923    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3845c16-0135-41b6-bc87-953357282374-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:14.752945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.752912    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3845c16-0135-41b6-bc87-953357282374-kube-api-access-wxwb6" pod "c3845c16-0135-41b6-bc87-953357282374" (UID: "c3845c16-0135-41b6-bc87-953357282374"). InnerVolumeSpecName "kube-api-access-wxwb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:37:14.753135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.753113    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3845c16-0135-41b6-bc87-953357282374-maas-api-xks-ca" pod "c3845c16-0135-41b6-bc87-953357282374" (UID: "c3845c16-0135-41b6-bc87-953357282374"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:37:14.853852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.853820    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3845c16-0135-41b6-bc87-953357282374-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:14.853852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:14.853846    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxwb6\" (UniqueName: \"kubernetes.io/projected/c3845c16-0135-41b6-bc87-953357282374-kube-api-access-wxwb6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:15.498297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:15.498264    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99" event={"ID":"c3845c16-0135-41b6-bc87-953357282374","Type":"ContainerDied","Data":"85f8f04a5b492e02a596fd3957736d63adf0849fbee69f703d58fd8de9cf5a87"}
Nov 02 14:37:15.498465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:15.498311    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85f8f04a5b492e02a596fd3957736d63adf0849fbee69f703d58fd8de9cf5a87"
Nov 02 14:37:15.498465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:15.498310    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"
Nov 02 14:37:18.443585 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:37:18.443549    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:37:20.142403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:20.142372    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:37:21.423370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.423338    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"]
Nov 02 14:37:21.424271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.424254    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c3845c16-0135-41b6-bc87-953357282374" containerName="revoke-keys"
Nov 02 14:37:21.424331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.424274    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3845c16-0135-41b6-bc87-953357282374" containerName="revoke-keys"
Nov 02 14:37:21.424464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.424454    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3845c16-0135-41b6-bc87-953357282374" containerName="revoke-keys"
Nov 02 14:37:21.428574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.428557    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.448684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.448660    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"]
Nov 02 14:37:21.456397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.456028    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:37:21.456397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.456041    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:37:21.456397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.456118    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:37:21.459760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.459738    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:37:21.599046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.599001    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgc4r\" (UniqueName: \"kubernetes.io/projected/0873c506-22e6-4c25-a9a2-492f47a11c70-kube-api-access-mgc4r\") pod \"maas-api-revoke-sub-fa5047d2-whqcz\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.599312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.599287    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa5047d2-whqcz\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.599462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.599436    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa5047d2-whqcz\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.701229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.701154    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa5047d2-whqcz\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.701373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.701286    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa5047d2-whqcz\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.701373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.701361    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgc4r\" (UniqueName: \"kubernetes.io/projected/0873c506-22e6-4c25-a9a2-492f47a11c70-kube-api-access-mgc4r\") pod \"maas-api-revoke-sub-fa5047d2-whqcz\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.701883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.701862    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa5047d2-whqcz\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.704091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.704065    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa5047d2-whqcz\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.726226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.726202    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgc4r\" (UniqueName: \"kubernetes.io/projected/0873c506-22e6-4c25-a9a2-492f47a11c70-kube-api-access-mgc4r\") pod \"maas-api-revoke-sub-fa5047d2-whqcz\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:21.744121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:21.744074    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:22.059043 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:37:22.059008    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0873c506_22e6_4c25_a9a2_492f47a11c70.slice/crio-26a890874d5932bb7de00f01ab86ba86e6d45dff667a42e7f90db6c335fdb5cb WatchSource:0}: Error finding container 26a890874d5932bb7de00f01ab86ba86e6d45dff667a42e7f90db6c335fdb5cb: Status 404 returned error can't find the container with id 26a890874d5932bb7de00f01ab86ba86e6d45dff667a42e7f90db6c335fdb5cb
Nov 02 14:37:22.062662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:22.062636    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"]
Nov 02 14:37:22.572973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:22.572932    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz" event={"ID":"0873c506-22e6-4c25-a9a2-492f47a11c70","Type":"ContainerStarted","Data":"5313c7d3eca089f0ae331f3f3212ac103cddd7ca5a406e78732824508d4da21b"}
Nov 02 14:37:22.573454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:22.573000    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz" event={"ID":"0873c506-22e6-4c25-a9a2-492f47a11c70","Type":"ContainerStarted","Data":"26a890874d5932bb7de00f01ab86ba86e6d45dff667a42e7f90db6c335fdb5cb"}
Nov 02 14:37:22.626118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:22.626069    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz" podStartSLOduration=1.626054302 podStartE2EDuration="1.626054302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:22.624189054 +0000 UTC m=+3233.375493494" watchObservedRunningTime="2026-11-02 14:37:22.626054302 +0000 UTC m=+3233.377358739"
Nov 02 14:37:23.627814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:23.627764    5034 generic.go:356] "Generic (PLEG): container finished" podID="0873c506-22e6-4c25-a9a2-492f47a11c70" containerID="5313c7d3eca089f0ae331f3f3212ac103cddd7ca5a406e78732824508d4da21b" exitCode=0
Nov 02 14:37:23.628818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:23.628757    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz" event={"ID":"0873c506-22e6-4c25-a9a2-492f47a11c70","Type":"ContainerDied","Data":"5313c7d3eca089f0ae331f3f3212ac103cddd7ca5a406e78732824508d4da21b"}
Nov 02 14:37:25.450456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.450424    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:25.578807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.575636    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-openshift-ca\") pod \"0873c506-22e6-4c25-a9a2-492f47a11c70\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") "
Nov 02 14:37:25.578807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.575764    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-xks-ca\") pod \"0873c506-22e6-4c25-a9a2-492f47a11c70\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") "
Nov 02 14:37:25.578807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.575897    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0873c506-22e6-4c25-a9a2-492f47a11c70-kube-api-access-mgc4r\" (UniqueName: \"kubernetes.io/projected/0873c506-22e6-4c25-a9a2-492f47a11c70-kube-api-access-mgc4r\") pod \"0873c506-22e6-4c25-a9a2-492f47a11c70\" (UID: \"0873c506-22e6-4c25-a9a2-492f47a11c70\") "
Nov 02 14:37:25.589846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.588707    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-openshift-ca" pod "0873c506-22e6-4c25-a9a2-492f47a11c70" (UID: "0873c506-22e6-4c25-a9a2-492f47a11c70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:37:25.658106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.655717    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0873c506-22e6-4c25-a9a2-492f47a11c70-kube-api-access-mgc4r" pod "0873c506-22e6-4c25-a9a2-492f47a11c70" (UID: "0873c506-22e6-4c25-a9a2-492f47a11c70"). InnerVolumeSpecName "kube-api-access-mgc4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:37:25.666992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.666958    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-xks-ca" pod "0873c506-22e6-4c25-a9a2-492f47a11c70" (UID: "0873c506-22e6-4c25-a9a2-492f47a11c70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:37:25.683767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.683735    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz" event={"ID":"0873c506-22e6-4c25-a9a2-492f47a11c70","Type":"ContainerDied","Data":"26a890874d5932bb7de00f01ab86ba86e6d45dff667a42e7f90db6c335fdb5cb"}
Nov 02 14:37:25.683984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.683966    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a890874d5932bb7de00f01ab86ba86e6d45dff667a42e7f90db6c335fdb5cb"
Nov 02 14:37:25.684198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.684182    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"
Nov 02 14:37:25.689257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.689235    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgc4r\" (UniqueName: \"kubernetes.io/projected/0873c506-22e6-4c25-a9a2-492f47a11c70-kube-api-access-mgc4r\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:25.689393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.689376    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:25.689509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:25.689496    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0873c506-22e6-4c25-a9a2-492f47a11c70-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:28.905358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.905322    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"]
Nov 02 14:37:28.906900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.906876    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0873c506-22e6-4c25-a9a2-492f47a11c70" containerName="revoke-keys"
Nov 02 14:37:28.906992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.906902    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0873c506-22e6-4c25-a9a2-492f47a11c70" containerName="revoke-keys"
Nov 02 14:37:28.907141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.907128    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0873c506-22e6-4c25-a9a2-492f47a11c70" containerName="revoke-keys"
Nov 02 14:37:28.912443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.912422    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:28.928170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.928144    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"]
Nov 02 14:37:28.934825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.934182    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:37:28.934825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.934255    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:37:28.934825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.934306    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:37:28.944341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.944321    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:37:28.973339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.973303    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8bgc\" (UniqueName: \"kubernetes.io/projected/37ad3ea2-1672-4f76-a3ad-01ec506622ea-kube-api-access-x8bgc\") pod \"maas-api-revoke-sub-2f33e918-dq555\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:28.973502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.973479    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f33e918-dq555\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:28.973825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:28.973805    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f33e918-dq555\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:29.075618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:29.075583    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f33e918-dq555\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:29.075850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:29.075707    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8bgc\" (UniqueName: \"kubernetes.io/projected/37ad3ea2-1672-4f76-a3ad-01ec506622ea-kube-api-access-x8bgc\") pod \"maas-api-revoke-sub-2f33e918-dq555\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:29.075922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:29.075859    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f33e918-dq555\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:29.076638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:29.076612    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f33e918-dq555\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:29.078704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:29.078678    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f33e918-dq555\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:29.101343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:29.101311    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8bgc\" (UniqueName: \"kubernetes.io/projected/37ad3ea2-1672-4f76-a3ad-01ec506622ea-kube-api-access-x8bgc\") pod \"maas-api-revoke-sub-2f33e918-dq555\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:29.231263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:29.231232    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:29.554817 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:37:29.554757    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ad3ea2_1672_4f76_a3ad_01ec506622ea.slice/crio-d544102ac596a3cd7c2bd96f1a51fa8a77ff35e7c8b894b9261d22b31d5dac40 WatchSource:0}: Error finding container d544102ac596a3cd7c2bd96f1a51fa8a77ff35e7c8b894b9261d22b31d5dac40: Status 404 returned error can't find the container with id d544102ac596a3cd7c2bd96f1a51fa8a77ff35e7c8b894b9261d22b31d5dac40
Nov 02 14:37:29.556759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:29.556734    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"]
Nov 02 14:37:29.727186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:29.727148    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555" event={"ID":"37ad3ea2-1672-4f76-a3ad-01ec506622ea","Type":"ContainerStarted","Data":"d544102ac596a3cd7c2bd96f1a51fa8a77ff35e7c8b894b9261d22b31d5dac40"}
Nov 02 14:37:30.737697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:30.737666    5034 generic.go:356] "Generic (PLEG): container finished" podID="37ad3ea2-1672-4f76-a3ad-01ec506622ea" containerID="f7b7cbef88e23becb89818f2667a4da039052a0e308d8272ca99b5a0b59725fe" exitCode=0
Nov 02 14:37:30.738111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:30.737750    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555" event={"ID":"37ad3ea2-1672-4f76-a3ad-01ec506622ea","Type":"ContainerDied","Data":"f7b7cbef88e23becb89818f2667a4da039052a0e308d8272ca99b5a0b59725fe"}
Nov 02 14:37:32.050196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.050169    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:32.150716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.150684    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-openshift-ca\") pod \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") "
Nov 02 14:37:32.150933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.150854    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-xks-ca\") pod \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") "
Nov 02 14:37:32.150990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.150965    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37ad3ea2-1672-4f76-a3ad-01ec506622ea-kube-api-access-x8bgc\" (UniqueName: \"kubernetes.io/projected/37ad3ea2-1672-4f76-a3ad-01ec506622ea-kube-api-access-x8bgc\") pod \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\" (UID: \"37ad3ea2-1672-4f76-a3ad-01ec506622ea\") "
Nov 02 14:37:32.151104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.151077    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-openshift-ca" pod "37ad3ea2-1672-4f76-a3ad-01ec506622ea" (UID: "37ad3ea2-1672-4f76-a3ad-01ec506622ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:37:32.151885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.151861    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:32.153091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.153061    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37ad3ea2-1672-4f76-a3ad-01ec506622ea-kube-api-access-x8bgc" pod "37ad3ea2-1672-4f76-a3ad-01ec506622ea" (UID: "37ad3ea2-1672-4f76-a3ad-01ec506622ea"). InnerVolumeSpecName "kube-api-access-x8bgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:37:32.153221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.153103    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-xks-ca" pod "37ad3ea2-1672-4f76-a3ad-01ec506622ea" (UID: "37ad3ea2-1672-4f76-a3ad-01ec506622ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:37:32.253990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.253962    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37ad3ea2-1672-4f76-a3ad-01ec506622ea-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:32.253990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.253988    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8bgc\" (UniqueName: \"kubernetes.io/projected/37ad3ea2-1672-4f76-a3ad-01ec506622ea-kube-api-access-x8bgc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:32.761431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.761402    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"
Nov 02 14:37:32.761608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.761405    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555" event={"ID":"37ad3ea2-1672-4f76-a3ad-01ec506622ea","Type":"ContainerDied","Data":"d544102ac596a3cd7c2bd96f1a51fa8a77ff35e7c8b894b9261d22b31d5dac40"}
Nov 02 14:37:32.761608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:32.761522    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d544102ac596a3cd7c2bd96f1a51fa8a77ff35e7c8b894b9261d22b31d5dac40"
Nov 02 14:37:39.823395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.823355    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"]
Nov 02 14:37:39.824296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.824280    5034 cpu_manager.go:414] "containerMap: removing container" podUID="37ad3ea2-1672-4f76-a3ad-01ec506622ea" containerName="revoke-keys"
Nov 02 14:37:39.824296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.824297    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37ad3ea2-1672-4f76-a3ad-01ec506622ea" containerName="revoke-keys"
Nov 02 14:37:39.824481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.824471    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="37ad3ea2-1672-4f76-a3ad-01ec506622ea" containerName="revoke-keys"
Nov 02 14:37:39.829896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.829876    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:39.856390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.856355    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:37:39.856719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.856691    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:37:39.861306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.860915    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:37:39.861488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.861047    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:37:39.874261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.874226    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"]
Nov 02 14:37:39.976286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.976244    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kl2c\" (UniqueName: \"kubernetes.io/projected/86b13f85-9472-40df-acfb-9fb13fb8b413-kube-api-access-5kl2c\") pod \"maas-api-revoke-sub-e55597c1-hwwql\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:39.976574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.976550    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e55597c1-hwwql\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:39.976684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:39.976669    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e55597c1-hwwql\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:40.032356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.032317    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"]
Nov 02 14:37:40.044049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.044010    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78649acf-s425c"]
Nov 02 14:37:40.078675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.078577    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e55597c1-hwwql\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:40.078889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.078685    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e55597c1-hwwql\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:40.078889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.078867    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kl2c\" (UniqueName: \"kubernetes.io/projected/86b13f85-9472-40df-acfb-9fb13fb8b413-kube-api-access-5kl2c\") pod \"maas-api-revoke-sub-e55597c1-hwwql\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:40.079864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.079836    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e55597c1-hwwql\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:40.081732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.081702    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e55597c1-hwwql\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:40.106872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.106835    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kl2c\" (UniqueName: \"kubernetes.io/projected/86b13f85-9472-40df-acfb-9fb13fb8b413-kube-api-access-5kl2c\") pod \"maas-api-revoke-sub-e55597c1-hwwql\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:40.146642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.146608    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:40.487947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.487911    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"]
Nov 02 14:37:40.493372 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:37:40.493340    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b13f85_9472_40df_acfb_9fb13fb8b413.slice/crio-9b5da4bb49369670fb6e8393783288496e7ad17ac58e97add4f82f2fc835e6d5 WatchSource:0}: Error finding container 9b5da4bb49369670fb6e8393783288496e7ad17ac58e97add4f82f2fc835e6d5: Status 404 returned error can't find the container with id 9b5da4bb49369670fb6e8393783288496e7ad17ac58e97add4f82f2fc835e6d5
Nov 02 14:37:40.999267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.999232    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql" event={"ID":"86b13f85-9472-40df-acfb-9fb13fb8b413","Type":"ContainerStarted","Data":"825ab3f3678cbb697e85b1a676f6a281435694ff46b607a8b5715d8274cdf034"}
Nov 02 14:37:40.999698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:40.999294    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql" event={"ID":"86b13f85-9472-40df-acfb-9fb13fb8b413","Type":"ContainerStarted","Data":"9b5da4bb49369670fb6e8393783288496e7ad17ac58e97add4f82f2fc835e6d5"}
Nov 02 14:37:41.046736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:41.046624    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql" podStartSLOduration=2.046609517 podStartE2EDuration="2.046609517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:37:41.042473213 +0000 UTC m=+3251.793777651" watchObservedRunningTime="2026-11-02 14:37:41.046609517 +0000 UTC m=+3251.797913955"
Nov 02 14:37:41.401317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:41.401234    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b615f723-80d6-4377-a4e4-70398bc891ac" path="/var/lib/kubelet/pods/b615f723-80d6-4377-a4e4-70398bc891ac/volumes"
Nov 02 14:37:42.015481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:42.015437    5034 generic.go:356] "Generic (PLEG): container finished" podID="86b13f85-9472-40df-acfb-9fb13fb8b413" containerID="825ab3f3678cbb697e85b1a676f6a281435694ff46b607a8b5715d8274cdf034" exitCode=0
Nov 02 14:37:42.016087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:42.015528    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql" event={"ID":"86b13f85-9472-40df-acfb-9fb13fb8b413","Type":"ContainerDied","Data":"825ab3f3678cbb697e85b1a676f6a281435694ff46b607a8b5715d8274cdf034"}
Nov 02 14:37:43.316548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.316508    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:43.356241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.356206    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b13f85-9472-40df-acfb-9fb13fb8b413-kube-api-access-5kl2c\" (UniqueName: \"kubernetes.io/projected/86b13f85-9472-40df-acfb-9fb13fb8b413-kube-api-access-5kl2c\") pod \"86b13f85-9472-40df-acfb-9fb13fb8b413\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") "
Nov 02 14:37:43.356449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.356259    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-openshift-ca\") pod \"86b13f85-9472-40df-acfb-9fb13fb8b413\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") "
Nov 02 14:37:43.356449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.356366    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-xks-ca\") pod \"86b13f85-9472-40df-acfb-9fb13fb8b413\" (UID: \"86b13f85-9472-40df-acfb-9fb13fb8b413\") "
Nov 02 14:37:43.356659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.356635    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-openshift-ca" pod "86b13f85-9472-40df-acfb-9fb13fb8b413" (UID: "86b13f85-9472-40df-acfb-9fb13fb8b413"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:37:43.357141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.357120    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:43.358684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.358658    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-xks-ca" pod "86b13f85-9472-40df-acfb-9fb13fb8b413" (UID: "86b13f85-9472-40df-acfb-9fb13fb8b413"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:37:43.358805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.358720    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b13f85-9472-40df-acfb-9fb13fb8b413-kube-api-access-5kl2c" pod "86b13f85-9472-40df-acfb-9fb13fb8b413" (UID: "86b13f85-9472-40df-acfb-9fb13fb8b413"). InnerVolumeSpecName "kube-api-access-5kl2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:37:43.458791 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.458737    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kl2c\" (UniqueName: \"kubernetes.io/projected/86b13f85-9472-40df-acfb-9fb13fb8b413-kube-api-access-5kl2c\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:43.458791 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:43.458772    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86b13f85-9472-40df-acfb-9fb13fb8b413-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:37:44.039091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:44.039059    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"
Nov 02 14:37:44.039322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:44.039086    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql" event={"ID":"86b13f85-9472-40df-acfb-9fb13fb8b413","Type":"ContainerDied","Data":"9b5da4bb49369670fb6e8393783288496e7ad17ac58e97add4f82f2fc835e6d5"}
Nov 02 14:37:44.039322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:37:44.039148    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b5da4bb49369670fb6e8393783288496e7ad17ac58e97add4f82f2fc835e6d5"
Nov 02 14:37:48.416050 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:37:48.416013    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:38:18.420310 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:18.420216    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:38:19.536453 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:19.536423    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-b174148781eb: secret "agc-cred-b174148781eb" not found
Nov 02 14:38:19.537246 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:19.536845    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb: secret "agc-cred-b174148781eb" not found
Nov 02 14:38:19.537246 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:19.536947    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials podName:fc067926-6d67-4ff5-aca8-666478c5fc6f nodeName:}" failed. No retries permitted until 2026-11-02 14:38:20.036925697 +0000 UTC m=+3290.788230127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" (UID: "fc067926-6d67-4ff5-aca8-666478c5fc6f") : secret "agc-cred-b174148781eb" not found
Nov 02 14:38:20.045770 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:20.045738    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-1e3a26fb0e59: secret "agc-cred-1e3a26fb0e59" not found
Nov 02 14:38:20.045770 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:20.045768    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-b174148781eb: secret "agc-cred-b174148781eb" not found
Nov 02 14:38:20.045996 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:20.045808    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb: [secret "agc-cred-1e3a26fb0e59" not found, secret "agc-cred-b174148781eb" not found]
Nov 02 14:38:20.045996 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:20.045876    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials podName:fc067926-6d67-4ff5-aca8-666478c5fc6f nodeName:}" failed. No retries permitted until 2026-11-02 14:38:21.045858173 +0000 UTC m=+3291.797162608 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" (UID: "fc067926-6d67-4ff5-aca8-666478c5fc6f") : [secret "agc-cred-1e3a26fb0e59" not found, secret "agc-cred-b174148781eb" not found]
Nov 02 14:38:20.963334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:20.963298    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"]
Nov 02 14:38:21.079468 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:21.079437    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-1e3a26fb0e59: secret "agc-cred-1e3a26fb0e59" not found
Nov 02 14:38:21.079468 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:21.079463    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-b174148781eb: secret "agc-cred-b174148781eb" not found
Nov 02 14:38:21.079468 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:21.079473    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb: [secret "agc-cred-1e3a26fb0e59" not found, secret "agc-cred-b174148781eb" not found]
Nov 02 14:38:21.079788 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:21.079540    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials podName:fc067926-6d67-4ff5-aca8-666478c5fc6f nodeName:}" failed. No retries permitted until 2026-11-02 14:38:23.079520496 +0000 UTC m=+3293.830824913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" (UID: "fc067926-6d67-4ff5-aca8-666478c5fc6f") : [secret "agc-cred-1e3a26fb0e59" not found, secret "agc-cred-b174148781eb" not found]
Nov 02 14:38:22.009856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:22.008767    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"]
Nov 02 14:38:23.128670 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:23.128639    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-1e3a26fb0e59: secret "agc-cred-1e3a26fb0e59" not found
Nov 02 14:38:23.128670 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:23.128667    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-b174148781eb: secret "agc-cred-b174148781eb" not found
Nov 02 14:38:23.128670 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:23.128680    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb: [secret "agc-cred-1e3a26fb0e59" not found, secret "agc-cred-b174148781eb" not found]
Nov 02 14:38:23.129319 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:23.128745    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials podName:fc067926-6d67-4ff5-aca8-666478c5fc6f nodeName:}" failed. No retries permitted until 2026-11-02 14:38:27.128731727 +0000 UTC m=+3297.880036147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" (UID: "fc067926-6d67-4ff5-aca8-666478c5fc6f") : [secret "agc-cred-1e3a26fb0e59" not found, secret "agc-cred-b174148781eb" not found]
Nov 02 14:38:25.963853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:25.963758    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm" podUID="591b724a-9b26-41d7-8085-eeab89e67d45" containerName="payload-processing" containerID="cri-o://071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec" gracePeriod=25
Nov 02 14:38:26.285310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.285280    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:38:26.386634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.386596    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-plugins-config-volume\") pod \"591b724a-9b26-41d7-8085-eeab89e67d45\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") "
Nov 02 14:38:26.386848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.386688    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-kube-api-access-5lrbf\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-kube-api-access-5lrbf\") pod \"591b724a-9b26-41d7-8085-eeab89e67d45\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") "
Nov 02 14:38:26.386848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.386827    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-provider-credentials\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-provider-credentials\") pod \"591b724a-9b26-41d7-8085-eeab89e67d45\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") "
Nov 02 14:38:26.387055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.386873    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-routing-overlay-volume\") pod \"591b724a-9b26-41d7-8085-eeab89e67d45\" (UID: \"591b724a-9b26-41d7-8085-eeab89e67d45\") "
Nov 02 14:38:26.387260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.387224    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-plugins-config-volume" pod "591b724a-9b26-41d7-8085-eeab89e67d45" (UID: "591b724a-9b26-41d7-8085-eeab89e67d45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:38:26.387355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.387250    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-routing-overlay-volume" pod "591b724a-9b26-41d7-8085-eeab89e67d45" (UID: "591b724a-9b26-41d7-8085-eeab89e67d45"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:38:26.387843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.387816    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:38:26.387970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.387850    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/591b724a-9b26-41d7-8085-eeab89e67d45-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:38:26.389419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.389386    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-provider-credentials" pod "591b724a-9b26-41d7-8085-eeab89e67d45" (UID: "591b724a-9b26-41d7-8085-eeab89e67d45"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:38:26.389969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.389944    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-kube-api-access-5lrbf" pod "591b724a-9b26-41d7-8085-eeab89e67d45" (UID: "591b724a-9b26-41d7-8085-eeab89e67d45"). InnerVolumeSpecName "kube-api-access-5lrbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:38:26.489474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.489441    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrbf\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-kube-api-access-5lrbf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:38:26.489474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.489470    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/591b724a-9b26-41d7-8085-eeab89e67d45-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:38:26.811380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.811340    5034 generic.go:356] "Generic (PLEG): container finished" podID="591b724a-9b26-41d7-8085-eeab89e67d45" containerID="071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec" exitCode=0
Nov 02 14:38:26.811575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.811396    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"
Nov 02 14:38:26.811575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.811418    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm" event={"ID":"591b724a-9b26-41d7-8085-eeab89e67d45","Type":"ContainerDied","Data":"071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec"}
Nov 02 14:38:26.811575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.811476    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm" event={"ID":"591b724a-9b26-41d7-8085-eeab89e67d45","Type":"ContainerDied","Data":"153d7ea7ed917f3571d69d99ead9c793e34fd3b1957666db102b969f293e7c0b"}
Nov 02 14:38:26.811575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.811500    5034 scope.go:122] "RemoveContainer" containerID="071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec"
Nov 02 14:38:26.827088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.827070    5034 scope.go:122] "RemoveContainer" containerID="071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec"
Nov 02 14:38:26.827430 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:26.827397    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec\": container with ID starting with 071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec not found: ID does not exist" containerID="071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec"
Nov 02 14:38:26.827541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.827439    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec"} err="failed to get container status \"071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec\": rpc error: code = NotFound desc = could not find container \"071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec\": container with ID starting with 071d5cce329cbc06f4aa5826876c3e47d005215aec4e7c5aa170bf298f2131ec not found: ID does not exist"
Nov 02 14:38:26.871344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.871318    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"]
Nov 02 14:38:26.883982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:26.883957    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bc34db/payload-processing-external-model-e2e-extmod-b-bc34db-9479w4nnm"]
Nov 02 14:38:27.010754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.010696    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" podUID="fc067926-6d67-4ff5-aca8-666478c5fc6f" containerName="payload-processing" containerID="cri-o://45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5" gracePeriod=25
Nov 02 14:38:27.200844 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:27.199949    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-1e3a26fb0e59: secret "agc-cred-1e3a26fb0e59" not found
Nov 02 14:38:27.200844 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:27.199981    5034 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-87efd3/agc-cred-b174148781eb: secret "agc-cred-b174148781eb" not found
Nov 02 14:38:27.200844 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:27.199994    5034 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb: [secret "agc-cred-1e3a26fb0e59" not found, secret "agc-cred-b174148781eb" not found]
Nov 02 14:38:27.200844 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:27.200063    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials podName:fc067926-6d67-4ff5-aca8-666478c5fc6f nodeName:}" failed. No retries permitted until 2026-11-02 14:38:35.200043924 +0000 UTC m=+3305.951348366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" (UID: "fc067926-6d67-4ff5-aca8-666478c5fc6f") : [secret "agc-cred-1e3a26fb0e59" not found, secret "agc-cred-b174148781eb" not found]
Nov 02 14:38:27.356275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.356249    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:38:27.399209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.399125    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591b724a-9b26-41d7-8085-eeab89e67d45" path="/var/lib/kubelet/pods/591b724a-9b26-41d7-8085-eeab89e67d45/volumes"
Nov 02 14:38:27.403045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.403010    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-plugins-config-volume\") pod \"fc067926-6d67-4ff5-aca8-666478c5fc6f\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") "
Nov 02 14:38:27.403208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.403118    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials\") pod \"fc067926-6d67-4ff5-aca8-666478c5fc6f\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") "
Nov 02 14:38:27.403386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.403347    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-plugins-config-volume" pod "fc067926-6d67-4ff5-aca8-666478c5fc6f" (UID: "fc067926-6d67-4ff5-aca8-666478c5fc6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:38:27.403386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.403373    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-routing-overlay-volume\") pod \"fc067926-6d67-4ff5-aca8-666478c5fc6f\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") "
Nov 02 14:38:27.403574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.403426    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-kube-api-access-jfprj\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-kube-api-access-jfprj\") pod \"fc067926-6d67-4ff5-aca8-666478c5fc6f\" (UID: \"fc067926-6d67-4ff5-aca8-666478c5fc6f\") "
Nov 02 14:38:27.403820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.403793    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-routing-overlay-volume" pod "fc067926-6d67-4ff5-aca8-666478c5fc6f" (UID: "fc067926-6d67-4ff5-aca8-666478c5fc6f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:38:27.404092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.404066    5034 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-routing-overlay-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:38:27.404213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.404094    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc067926-6d67-4ff5-aca8-666478c5fc6f-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:38:27.405385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.405356    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials" pod "fc067926-6d67-4ff5-aca8-666478c5fc6f" (UID: "fc067926-6d67-4ff5-aca8-666478c5fc6f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:38:27.405656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.405630    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-kube-api-access-jfprj" pod "fc067926-6d67-4ff5-aca8-666478c5fc6f" (UID: "fc067926-6d67-4ff5-aca8-666478c5fc6f"). InnerVolumeSpecName "kube-api-access-jfprj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:38:27.505329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.505297    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfprj\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-kube-api-access-jfprj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:38:27.505329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.505324    5034 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc067926-6d67-4ff5-aca8-666478c5fc6f-provider-credentials\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:38:27.823876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.823841    5034 generic.go:356] "Generic (PLEG): container finished" podID="fc067926-6d67-4ff5-aca8-666478c5fc6f" containerID="45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5" exitCode=0
Nov 02 14:38:27.824080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.823893    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"
Nov 02 14:38:27.824080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.823924    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" event={"ID":"fc067926-6d67-4ff5-aca8-666478c5fc6f","Type":"ContainerDied","Data":"45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5"}
Nov 02 14:38:27.824080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.823982    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb" event={"ID":"fc067926-6d67-4ff5-aca8-666478c5fc6f","Type":"ContainerDied","Data":"3c22a45342a77abc254f4738a33b3d56fdac5900003f438eeb992c95b2855e8d"}
Nov 02 14:38:27.824080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.824009    5034 scope.go:122] "RemoveContainer" containerID="45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5"
Nov 02 14:38:27.838248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.838227    5034 scope.go:122] "RemoveContainer" containerID="45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5"
Nov 02 14:38:27.838529 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:27.838508    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5\": container with ID starting with 45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5 not found: ID does not exist" containerID="45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5"
Nov 02 14:38:27.838617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.838543    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5"} err="failed to get container status \"45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5\": rpc error: code = NotFound desc = could not find container \"45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5\": container with ID starting with 45715417b651ab64a31a0089efa6eae277930239a25a22b3232bb625d09dbaf5 not found: ID does not exist"
Nov 02 14:38:27.881246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.881224    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"]
Nov 02 14:38:27.886870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:27.886850    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-87efd3/payload-processing-external-model-e2e-extmod-a-87efd3-85759d6jb"]
Nov 02 14:38:29.399348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.399313    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc067926-6d67-4ff5-aca8-666478c5fc6f" path="/var/lib/kubelet/pods/fc067926-6d67-4ff5-aca8-666478c5fc6f/volumes"
Nov 02 14:38:29.845403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.845363    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"]
Nov 02 14:38:29.846757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.846735    5034 cpu_manager.go:414] "containerMap: removing container" podUID="591b724a-9b26-41d7-8085-eeab89e67d45" containerName="payload-processing"
Nov 02 14:38:29.846846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.846761    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591b724a-9b26-41d7-8085-eeab89e67d45" containerName="payload-processing"
Nov 02 14:38:29.846846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.846795    5034 cpu_manager.go:414] "containerMap: removing container" podUID="fc067926-6d67-4ff5-aca8-666478c5fc6f" containerName="payload-processing"
Nov 02 14:38:29.846846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.846803    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc067926-6d67-4ff5-aca8-666478c5fc6f" containerName="payload-processing"
Nov 02 14:38:29.846993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.846915    5034 cpu_manager.go:414] "containerMap: removing container" podUID="86b13f85-9472-40df-acfb-9fb13fb8b413" containerName="revoke-keys"
Nov 02 14:38:29.846993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.846925    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b13f85-9472-40df-acfb-9fb13fb8b413" containerName="revoke-keys"
Nov 02 14:38:29.847143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.847130    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b13f85-9472-40df-acfb-9fb13fb8b413" containerName="revoke-keys"
Nov 02 14:38:29.847184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.847147    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="591b724a-9b26-41d7-8085-eeab89e67d45" containerName="payload-processing"
Nov 02 14:38:29.847184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.847158    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc067926-6d67-4ff5-aca8-666478c5fc6f" containerName="payload-processing"
Nov 02 14:38:29.853016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.852988    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:29.880283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.880244    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c8f647-openshift-default-dockercfg-9w5hz\""
Nov 02 14:38:29.897161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.897122    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"]
Nov 02 14:38:29.944687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.944651    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-certs\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:29.944902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.944878    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-data\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:29.944975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.944933    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-envoy\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:29.945134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.945113    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istiod-ca-cert\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:29.945213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.945194    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-token\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:29.945310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.945286    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-credential-socket\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:29.945359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.945348    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-socket\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:29.945449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.945427    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgkjf\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-kube-api-access-dgkjf\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:29.945667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:29.945648    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-podinfo\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.244872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.154868    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-certs\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.266109    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-data\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.266215    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-envoy\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.266388    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istiod-ca-cert\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.266475    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-token\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.266559    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-credential-socket\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.266630    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-socket\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.266667    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgkjf\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-kube-api-access-dgkjf\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.266765    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-podinfo\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.273560    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istiod-ca-cert\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.244766    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-certs\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.273847    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-data\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.283642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.276407    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-socket\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.313861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.289519    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-credential-socket\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.313861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.295466    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-envoy\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.313861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.301089    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-token\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.313861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.303381    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-podinfo\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.316519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.316466    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgkjf\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-kube-api-access-dgkjf\") pod \"e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") " pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.467731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.467691    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:30.765723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.765692    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:38:30.767927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.767902    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:38:30.923802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.922922    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"]
Nov 02 14:38:30.933000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.932726    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:38:30.933000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.932867    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:38:30.933000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:30.932982    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:38:31.932388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:31.932343    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" event={"ID":"e4e4ace3-b064-47cc-a5f0-031dac8babf2","Type":"ContainerStarted","Data":"c69652c3bf2220dcb0d0e57aeac6982ddada3f044746d82c0efbb0dfc3eb4ad2"}
Nov 02 14:38:31.932962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:31.932448    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" event={"ID":"e4e4ace3-b064-47cc-a5f0-031dac8babf2","Type":"ContainerStarted","Data":"fa0f00f494d86b27ea1daba568d071c04cc2417e3280a69ad94b6ccbef255add"}
Nov 02 14:38:32.018310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:32.018252    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" podStartSLOduration=3.018238262 podStartE2EDuration="3.018238262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:32.010978523 +0000 UTC m=+3302.762282959" watchObservedRunningTime="2026-11-02 14:38:32.018238262 +0000 UTC m=+3302.769542699"
Nov 02 14:38:32.468512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:32.468474    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:32.502115 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:32.502071    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:32.943638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:32.943598    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:32.944758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:32.944735    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:38:36.707441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:36.707406    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"]
Nov 02 14:38:36.715848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:36.715823    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:36.751520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:36.751480    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c8f647\""
Nov 02 14:38:36.757524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:36.757477    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"]
Nov 02 14:38:36.892130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:36.892088    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d5f06f-cab1-4546-a546-95fc619bce7b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg\" (UID: \"02d5f06f-cab1-4546-a546-95fc619bce7b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:36.922749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:36.922715    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpkn8\" (UniqueName: \"kubernetes.io/projected/02d5f06f-cab1-4546-a546-95fc619bce7b-kube-api-access-tpkn8\") pod \"payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg\" (UID: \"02d5f06f-cab1-4546-a546-95fc619bce7b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:36.923244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:36.892703    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c8f647-dockercfg-mxgtc\""
Nov 02 14:38:37.115873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:37.099050    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d5f06f-cab1-4546-a546-95fc619bce7b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg\" (UID: \"02d5f06f-cab1-4546-a546-95fc619bce7b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:37.115873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:37.099307    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpkn8\" (UniqueName: \"kubernetes.io/projected/02d5f06f-cab1-4546-a546-95fc619bce7b-kube-api-access-tpkn8\") pod \"payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg\" (UID: \"02d5f06f-cab1-4546-a546-95fc619bce7b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:37.115873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:37.100583    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d5f06f-cab1-4546-a546-95fc619bce7b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg\" (UID: \"02d5f06f-cab1-4546-a546-95fc619bce7b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:37.168033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:37.167261    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpkn8\" (UniqueName: \"kubernetes.io/projected/02d5f06f-cab1-4546-a546-95fc619bce7b-kube-api-access-tpkn8\") pod \"payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg\" (UID: \"02d5f06f-cab1-4546-a546-95fc619bce7b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:37.351904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:37.348557    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:37.707761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:37.707736    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"]
Nov 02 14:38:37.918431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:37.918185    5034 scope.go:122] "RemoveContainer" containerID="18de6e94ff659f4873165d735802390e67931d0cf657036021142e622342a65a"
Nov 02 14:38:37.951924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:37.951898    5034 scope.go:122] "RemoveContainer" containerID="53f809c2cb0850f1531e1e1e0e1609740b512c8bd7bd37a72f8d0ec492600c66"
Nov 02 14:38:37.965051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:37.965027    5034 scope.go:122] "RemoveContainer" containerID="ab95953f506f017cbbd34f649e6e332c71b13a093e81bc0268729747a0768aaf"
Nov 02 14:38:38.181907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:38.181868    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg" event={"ID":"02d5f06f-cab1-4546-a546-95fc619bce7b","Type":"ContainerStarted","Data":"0f5b02fc3d4b8a0fd561addf7c17321528ad0349a81b74ec4e266cef703388bc"}
Nov 02 14:38:38.182091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:38.181937    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg" event={"ID":"02d5f06f-cab1-4546-a546-95fc619bce7b","Type":"ContainerStarted","Data":"d47234ddcb166ad61f59a2f6c1e8ad6a567e755202587c52f5a09a05a77bd723"}
Nov 02 14:38:38.182091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:38.181971    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:38.242805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:38.242739    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg" podStartSLOduration=2.242724839 podStartE2EDuration="2.242724839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:38.240522343 +0000 UTC m=+3308.991826782" watchObservedRunningTime="2026-11-02 14:38:38.242724839 +0000 UTC m=+3308.994029276"
Nov 02 14:38:41.743136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:41.741170    5034 patch_prober.go:29] interesting pod/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": read tcp 10.132.0.2:36498->10.132.1.96:15021: read: connection reset by peer" start-of-body=
Nov 02 14:38:41.743136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:41.741289    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": read tcp 10.132.0.2:36498->10.132.1.96:15021: read: connection reset by peer"
Nov 02 14:38:42.242303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:42.242262    5034 generic.go:356] "Generic (PLEG): container finished" podID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerID="116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee" exitCode=0
Nov 02 14:38:42.242498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:42.242338    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" event={"ID":"2f1bf78b-d6af-48b8-b6c7-44c8f4694240","Type":"ContainerDied","Data":"116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee"}
Nov 02 14:38:42.242832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:42.242814    5034 scope.go:122] "RemoveContainer" containerID="116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee"
Nov 02 14:38:42.244880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:42.244849    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:38:42.244970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:42.244920    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:38:42.244970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:42.244958    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:38:43.253588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.253556    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" event={"ID":"2f1bf78b-d6af-48b8-b6c7-44c8f4694240","Type":"ContainerStarted","Data":"4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa"}
Nov 02 14:38:43.383801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.378509    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"]
Nov 02 14:38:43.444188 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.444151    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:38:43.555064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.459812    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwnjf\" (UniqueName: \"kubernetes.io/projected/ca604bd3-98dc-408b-8c70-4ffc24eec28f-kube-api-access-qwnjf\") pod \"payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v\" (UID: \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:38:43.561746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.561712    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca604bd3-98dc-408b-8c70-4ffc24eec28f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v\" (UID: \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:38:43.716419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.669857    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwnjf\" (UniqueName: \"kubernetes.io/projected/ca604bd3-98dc-408b-8c70-4ffc24eec28f-kube-api-access-qwnjf\") pod \"payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v\" (UID: \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:38:43.735684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.717087    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca604bd3-98dc-408b-8c70-4ffc24eec28f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v\" (UID: \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:38:43.790893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.790861    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cbde78-dockercfg-cg4pv\""
Nov 02 14:38:43.800843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.790950    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cbde78\""
Nov 02 14:38:43.840625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.840552    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca604bd3-98dc-408b-8c70-4ffc24eec28f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v\" (UID: \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:38:43.842604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.842518    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwnjf\" (UniqueName: \"kubernetes.io/projected/ca604bd3-98dc-408b-8c70-4ffc24eec28f-kube-api-access-qwnjf\") pod \"payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v\" (UID: \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:38:43.858381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.847675    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"]
Nov 02 14:38:43.892856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:43.892822    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:38:44.117120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:44.116841    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:38:44.126352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:44.125443    5034 patch_prober.go:29] interesting pod/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 14:38:44.126352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:44.125498    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 14:38:44.356451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:44.356426    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"]
Nov 02 14:38:45.109871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:45.108383    5034 patch_prober.go:29] interesting pod/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 14:38:45.109871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:45.108444    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 14:38:45.294803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:45.280438    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v" event={"ID":"ca604bd3-98dc-408b-8c70-4ffc24eec28f","Type":"ContainerStarted","Data":"c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c"}
Nov 02 14:38:45.294803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:45.280546    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v" event={"ID":"ca604bd3-98dc-408b-8c70-4ffc24eec28f","Type":"ContainerStarted","Data":"995b14e3bff9ea171f205807fb03cfcf948f8611e60fb0a99f10796cf3efc285"}
Nov 02 14:38:45.294803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:45.280881    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:38:45.330692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:45.330631    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v" podStartSLOduration=2.330616106 podStartE2EDuration="2.330616106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:38:45.327748244 +0000 UTC m=+3316.079052683" watchObservedRunningTime="2026-11-02 14:38:45.330616106 +0000 UTC m=+3316.081920545"
Nov 02 14:38:46.105804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:46.105117    5034 patch_prober.go:29] interesting pod/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 14:38:46.105804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:46.105176    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 14:38:47.108010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:47.107935    5034 patch_prober.go:29] interesting pod/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 14:38:47.108010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:47.107990    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 14:38:48.107572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:48.107523    5034 patch_prober.go:29] interesting pod/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused" start-of-body=
Nov 02 14:38:48.107889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:48.107850    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.96:15021/healthz/ready\": dial tcp 10.132.1.96:15021: connect: connection refused"
Nov 02 14:38:48.433563 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:38:48.433486    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:38:49.107143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:49.107107    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:38:49.107617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:49.107591    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:38:49.108470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:49.108450    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:38:49.201440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:49.201411    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:38:56.381802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:38:56.378155    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:39:01.571229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:01.571197    5034 generic.go:356] "Generic (PLEG): container finished" podID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerID="74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379" exitCode=0
Nov 02 14:39:01.571670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:01.571276    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" event={"ID":"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979","Type":"ContainerDied","Data":"74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379"}
Nov 02 14:39:01.571828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:01.571771    5034 scope.go:122] "RemoveContainer" containerID="74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379"
Nov 02 14:39:01.574024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:01.573994    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:39:01.574145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:01.574081    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:39:01.574145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:01.574134    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:39:02.594804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:02.592248    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" event={"ID":"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979","Type":"ContainerStarted","Data":"e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b"}
Nov 02 14:39:02.955201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:02.954993    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:39:02.963799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:02.959874    5034 patch_prober.go:29] interesting pod/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Nov 02 14:39:02.963799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:02.959930    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Nov 02 14:39:03.955291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:03.955260    5034 patch_prober.go:29] interesting pod/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Nov 02 14:39:03.955707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:03.955319    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Nov 02 14:39:04.891804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:04.890870    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:39:04.969978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:04.965478    5034 patch_prober.go:29] interesting pod/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Nov 02 14:39:04.969978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:04.965547    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Nov 02 14:39:05.955843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:05.955806    5034 patch_prober.go:29] interesting pod/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Nov 02 14:39:05.956030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:05.955878    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Nov 02 14:39:06.955609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:06.955575    5034 patch_prober.go:29] interesting pod/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Nov 02 14:39:06.956321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:06.955627    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Nov 02 14:39:07.955235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:07.955194    5034 patch_prober.go:29] interesting pod/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body=
Nov 02 14:39:07.955445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:07.955261    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused"
Nov 02 14:39:08.960905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:08.960873    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:39:08.962118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:08.962093    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:39:11.549896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.549855    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"]
Nov 02 14:39:11.559147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.559122    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.579993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.579964    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"]
Nov 02 14:39:11.586926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.586906    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-09d1a5-openshift-default-dockercfg-7r6q6\""
Nov 02 14:39:11.730178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.730140    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsrm6\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-kube-api-access-nsrm6\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.730377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.730258    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b62861-322e-48a0-938d-539acb06cbcf-istio-podinfo\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.730377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.730324    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-credential-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.730377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.730350    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-certs\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.730579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.730466    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-istio-token\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.730579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.730506    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b62861-322e-48a0-938d-539acb06cbcf-istiod-ca-cert\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.730698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.730655    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-envoy\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.730853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.730836    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.730936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.730914    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-data\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.831985 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.831908    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsrm6\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-kube-api-access-nsrm6\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832065    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b62861-322e-48a0-938d-539acb06cbcf-istio-podinfo\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832167    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-credential-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832206    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-certs\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832310    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-istio-token\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832354    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b62861-322e-48a0-938d-539acb06cbcf-istiod-ca-cert\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832472    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-envoy\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832638    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-credential-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832646    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832709    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-certs\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.832837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832748    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-data\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.833008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.832979    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-socket\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.833065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.833047    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-data\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.833272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.833248    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b62861-322e-48a0-938d-539acb06cbcf-istiod-ca-cert\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.834718 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.834695    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-envoy\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.834874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.834844    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b62861-322e-48a0-938d-539acb06cbcf-istio-podinfo\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.855949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.855923    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-istio-token\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.856052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.856039    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsrm6\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-kube-api-access-nsrm6\") pod \"e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") " pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:11.874941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:11.874915    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:12.203244 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:39:12.203207    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b62861_322e_48a0_938d_539acb06cbcf.slice/crio-65d1c575608ee4d529916df7329af659366a4c102cdd3ca6d1b876911909f33d WatchSource:0}: Error finding container 65d1c575608ee4d529916df7329af659366a4c102cdd3ca6d1b876911909f33d: Status 404 returned error can't find the container with id 65d1c575608ee4d529916df7329af659366a4c102cdd3ca6d1b876911909f33d
Nov 02 14:39:12.207650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:12.207607    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:39:12.207773 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:12.207699    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:39:12.207773 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:12.207753    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:39:12.207913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:12.207884    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"]
Nov 02 14:39:12.767802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:12.767754    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" event={"ID":"d4b62861-322e-48a0-938d-539acb06cbcf","Type":"ContainerStarted","Data":"ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8"}
Nov 02 14:39:12.768203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:12.767828    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" event={"ID":"d4b62861-322e-48a0-938d-539acb06cbcf","Type":"ContainerStarted","Data":"65d1c575608ee4d529916df7329af659366a4c102cdd3ca6d1b876911909f33d"}
Nov 02 14:39:12.821268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:12.821199    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" podStartSLOduration=1.821179783 podStartE2EDuration="1.821179783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:12.815265996 +0000 UTC m=+3343.566570434" watchObservedRunningTime="2026-11-02 14:39:12.821179783 +0000 UTC m=+3343.572484226"
Nov 02 14:39:12.887802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:12.878891    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:13.897802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:13.896872    5034 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 02 14:39:13.897802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:13.896943    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 02 14:39:14.888151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:14.888117    5034 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 02 14:39:14.888439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:14.888410    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 02 14:39:15.039963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:15.039930    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"]
Nov 02 14:39:15.051892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:15.051861    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce4cb093-9vs6c"]
Nov 02 14:39:15.448030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:15.447992    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd28dcd-1c18-43ab-924d-0abbb0df7789" path="/var/lib/kubelet/pods/acd28dcd-1c18-43ab-924d-0abbb0df7789/volumes"
Nov 02 14:39:15.877140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:15.877105    5034 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 02 14:39:15.877432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:15.877400    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 02 14:39:16.886802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:16.880410    5034 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 02 14:39:16.886802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:16.880464    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 02 14:39:17.101799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:17.096090    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"]
Nov 02 14:39:17.128609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:17.128573    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc09e0ea-8qgg6"]
Nov 02 14:39:17.416248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:17.416205    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f111d75b-bba3-426f-b366-f471654ae09f" path="/var/lib/kubelet/pods/f111d75b-bba3-426f-b366-f471654ae09f/volumes"
Nov 02 14:39:17.878123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:17.878087    5034 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body=
Nov 02 14:39:17.899958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:17.899913    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused"
Nov 02 14:39:18.417463 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:39:18.417417    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:39:18.880570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:18.880544    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:18.881048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:18.881025    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:18.881642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:18.881617    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:39:30.102982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:30.102826    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"]
Nov 02 14:39:30.196503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:30.188233    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-248b3f02-6wn4p"]
Nov 02 14:39:31.425937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:31.425902    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b93c95ca-53f4-4076-98b9-c156cb0d7d9b" path="/var/lib/kubelet/pods/b93c95ca-53f4-4076-98b9-c156cb0d7d9b/volumes"
Nov 02 14:39:44.128384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.128295    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"]
Nov 02 14:39:44.134884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.134862    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.150684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.150664    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d39fa6-openshift-default-dockercfg-8rt29\""
Nov 02 14:39:44.170371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.170327    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"]
Nov 02 14:39:44.192090 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.192062    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.192540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.192463    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istiod-ca-cert\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.193988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.193968    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-podinfo\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.194236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.194216    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-certs\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.194393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.194375    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-credential-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.194544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.194524    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-data\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.194771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.194753    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-token\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.194960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.194933    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-envoy\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.195081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.194993    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7pwp\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-kube-api-access-v7pwp\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296331    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-certs\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296384    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-credential-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296438    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-data\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296601    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-token\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296649    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-envoy\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296701    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7pwp\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-kube-api-access-v7pwp\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296799    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296846    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-certs\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296847    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-credential-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.296981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.296916    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-data\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.297089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.297030    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istiod-ca-cert\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.297151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.297107    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-podinfo\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.297151 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.297113    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-socket\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.297648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.297624    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istiod-ca-cert\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.299552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.299527    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-envoy\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.299662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.299611    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-podinfo\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.323414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.323386    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-token\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.323659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.323640    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7pwp\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-kube-api-access-v7pwp\") pod \"e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") " pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.450505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.450431    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:44.789159 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:39:44.789125    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd9dc24_44e7_4c1e_ac79_e45383aaab73.slice/crio-bf9d20891b31cd2efe5578a50c970297a33d1afadbe90d0962347cd3d52099b5 WatchSource:0}: Error finding container bf9d20891b31cd2efe5578a50c970297a33d1afadbe90d0962347cd3d52099b5: Status 404 returned error can't find the container with id bf9d20891b31cd2efe5578a50c970297a33d1afadbe90d0962347cd3d52099b5
Nov 02 14:39:44.790941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.790911    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"]
Nov 02 14:39:44.792573 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.792536    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:39:44.792667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.792603    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:39:44.792667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:44.792635    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:39:45.281833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:45.278818    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" event={"ID":"bfd9dc24-44e7-4c1e-ac79-e45383aaab73","Type":"ContainerStarted","Data":"e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142"}
Nov 02 14:39:45.281833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:45.278887    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" event={"ID":"bfd9dc24-44e7-4c1e-ac79-e45383aaab73","Type":"ContainerStarted","Data":"bf9d20891b31cd2efe5578a50c970297a33d1afadbe90d0962347cd3d52099b5"}
Nov 02 14:39:45.343123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:45.343005    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" podStartSLOduration=1.342990519 podStartE2EDuration="1.342990519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:39:45.338398468 +0000 UTC m=+3376.089702909" watchObservedRunningTime="2026-11-02 14:39:45.342990519 +0000 UTC m=+3376.094294957"
Nov 02 14:39:45.450670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:45.450640    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:46.454677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:46.454633    5034 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 14:39:46.455327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:46.455292    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 14:39:47.451083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:47.451053    5034 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 14:39:47.451266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:47.451100    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 14:39:48.431584 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:39:48.431551    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:39:48.451002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:48.450967    5034 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 14:39:48.451171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:48.451038    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 14:39:49.451914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:49.451882    5034 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 14:39:49.452533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:49.452504    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 14:39:50.451239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:50.451199    5034 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 14:39:50.451442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:50.451259    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 14:39:51.451285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:51.451252    5034 patch_prober.go:29] interesting pod/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused" start-of-body=
Nov 02 14:39:51.451687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:51.451305    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.110:15021/healthz/ready\": dial tcp 10.132.1.110:15021: connect: connection refused"
Nov 02 14:39:52.680808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:52.680762    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:52.682096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:52.682063    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:39:52.683362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:39:52.683339    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:40:18.416518 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:40:18.416477    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:40:25.711357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:25.711322    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"]
Nov 02 14:40:25.711947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:25.711596    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy" containerID="cri-o://f1089380c61b5aefd94ea3c5091a65b74f6e9a8b593b069578e2dc26144c903c" gracePeriod=30
Nov 02 14:40:31.013290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.013260    5034 generic.go:356] "Generic (PLEG): container finished" podID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerID="f1089380c61b5aefd94ea3c5091a65b74f6e9a8b593b069578e2dc26144c903c" exitCode=0
Nov 02 14:40:31.013808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.013334    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" event={"ID":"23cc2e99-f558-4e2c-be3a-ed110bebf2e9","Type":"ContainerDied","Data":"f1089380c61b5aefd94ea3c5091a65b74f6e9a8b593b069578e2dc26144c903c"}
Nov 02 14:40:31.013808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.013405    5034 scope.go:122] "RemoveContainer" containerID="011a8d5633394df0e612db47dd7fe4c68859667ca1e4c756135c79f580b17a6e"
Nov 02 14:40:31.045616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.045594    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:40:31.099982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.099957    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-token\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-token\") pod \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") "
Nov 02 14:40:31.100192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100013    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-podinfo\") pod \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") "
Nov 02 14:40:31.100192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100059    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-data\") pod \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") "
Nov 02 14:40:31.100316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100275    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-certs\") pod \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") "
Nov 02 14:40:31.100375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100353    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-socket\") pod \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") "
Nov 02 14:40:31.100435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100396    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-data" pod "23cc2e99-f558-4e2c-be3a-ed110bebf2e9" (UID: "23cc2e99-f558-4e2c-be3a-ed110bebf2e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:31.100435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100406    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-kube-api-access-z7p9t\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-kube-api-access-z7p9t\") pod \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") "
Nov 02 14:40:31.100541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100448    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istiod-ca-cert\") pod \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") "
Nov 02 14:40:31.100612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100542    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-credential-socket\") pod \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") "
Nov 02 14:40:31.100612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100542    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-certs" pod "23cc2e99-f558-4e2c-be3a-ed110bebf2e9" (UID: "23cc2e99-f558-4e2c-be3a-ed110bebf2e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:31.100727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100622    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-socket" pod "23cc2e99-f558-4e2c-be3a-ed110bebf2e9" (UID: "23cc2e99-f558-4e2c-be3a-ed110bebf2e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:31.100727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100657    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-envoy\") pod \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\" (UID: \"23cc2e99-f558-4e2c-be3a-ed110bebf2e9\") "
Nov 02 14:40:31.100923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100901    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-credential-socket" pod "23cc2e99-f558-4e2c-be3a-ed110bebf2e9" (UID: "23cc2e99-f558-4e2c-be3a-ed110bebf2e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:31.100986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.100820    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istiod-ca-cert" pod "23cc2e99-f558-4e2c-be3a-ed110bebf2e9" (UID: "23cc2e99-f558-4e2c-be3a-ed110bebf2e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:40:31.101407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.101388    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:31.101499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.101411    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:31.101499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.101424    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:31.101499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.101436    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:31.101499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.101449    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:31.102549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.102519    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-token" pod "23cc2e99-f558-4e2c-be3a-ed110bebf2e9" (UID: "23cc2e99-f558-4e2c-be3a-ed110bebf2e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:31.102716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.102692    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-podinfo" pod "23cc2e99-f558-4e2c-be3a-ed110bebf2e9" (UID: "23cc2e99-f558-4e2c-be3a-ed110bebf2e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:40:31.103110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.103082    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-envoy" pod "23cc2e99-f558-4e2c-be3a-ed110bebf2e9" (UID: "23cc2e99-f558-4e2c-be3a-ed110bebf2e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:31.103285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.103262    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-kube-api-access-z7p9t" pod "23cc2e99-f558-4e2c-be3a-ed110bebf2e9" (UID: "23cc2e99-f558-4e2c-be3a-ed110bebf2e9"). InnerVolumeSpecName "kube-api-access-z7p9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:31.203947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.203907    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:31.203947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.203941    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:31.203947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.203956    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:31.204272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:31.203969    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7p9t\" (UniqueName: \"kubernetes.io/projected/23cc2e99-f558-4e2c-be3a-ed110bebf2e9-kube-api-access-z7p9t\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:32.024934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:32.024907    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"
Nov 02 14:40:32.025406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:32.024950    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k" event={"ID":"23cc2e99-f558-4e2c-be3a-ed110bebf2e9","Type":"ContainerDied","Data":"6fc004d43f3e2d8b6921e76c297431c21a6a38745a2de7324b7187546060c8df"}
Nov 02 14:40:32.025406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:32.025012    5034 scope.go:122] "RemoveContainer" containerID="f1089380c61b5aefd94ea3c5091a65b74f6e9a8b593b069578e2dc26144c903c"
Nov 02 14:40:32.143330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:32.143302    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"]
Nov 02 14:40:32.148294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:32.148262    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ce0f55-openshift-default-7486f976d9-lkg2k"]
Nov 02 14:40:33.404277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:33.404242    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" path="/var/lib/kubelet/pods/23cc2e99-f558-4e2c-be3a-ed110bebf2e9/volumes"
Nov 02 14:40:38.096800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:38.096459    5034 scope.go:122] "RemoveContainer" containerID="85cbd5235a0ff15ec84172a39d583f158bb396ba1f1369f64dc27284bcb95236"
Nov 02 14:40:38.117129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:38.117068    5034 scope.go:122] "RemoveContainer" containerID="f0a127bf3afe8822ff556b82b095a250aac03802a81a83f866647f303dabc7c5"
Nov 02 14:40:38.131630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:38.131612    5034 scope.go:122] "RemoveContainer" containerID="1eaf8a35aaf1b137b4d35626b4a1a1396af6e229c5e547dc4df2e2d0851b4c15"
Nov 02 14:40:39.025862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:39.025826    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"]
Nov 02 14:40:39.026071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:39.026030    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy" containerID="cri-o://eb05fabc6444e265685056cf0dad1fd2bf1db0a8c63c636efccdf73b112371ae" gracePeriod=30
Nov 02 14:40:44.643307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.643275    5034 generic.go:356] "Generic (PLEG): container finished" podID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerID="eb05fabc6444e265685056cf0dad1fd2bf1db0a8c63c636efccdf73b112371ae" exitCode=0
Nov 02 14:40:44.643819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.643359    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" event={"ID":"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920","Type":"ContainerDied","Data":"eb05fabc6444e265685056cf0dad1fd2bf1db0a8c63c636efccdf73b112371ae"}
Nov 02 14:40:44.782046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.777092    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:40:44.888849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.888509    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istiod-ca-cert\") pod \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") "
Nov 02 14:40:44.888849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.888587    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-token\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-token\") pod \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") "
Nov 02 14:40:44.888849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.888645    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-envoy\") pod \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") "
Nov 02 14:40:44.888849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.888754    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-socket\") pod \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") "
Nov 02 14:40:44.888849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.888810    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-certs\") pod \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") "
Nov 02 14:40:44.889311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.888970    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-data\") pod \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") "
Nov 02 14:40:44.889311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.889010    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-podinfo\") pod \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") "
Nov 02 14:40:44.889311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.889036    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-kube-api-access-m2dnl\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-kube-api-access-m2dnl\") pod \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") "
Nov 02 14:40:44.889311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.889116    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-credential-socket\") pod \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\" (UID: \"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920\") "
Nov 02 14:40:44.890521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.890489    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-socket" pod "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" (UID: "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:44.890828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.890808    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istiod-ca-cert" pod "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" (UID: "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:40:44.891179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.891156    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-credential-socket" pod "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" (UID: "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:44.891486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.891464    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-certs" pod "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" (UID: "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:44.891654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.891635    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-data" pod "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" (UID: "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:44.897329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.897299    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-token" pod "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" (UID: "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:44.902078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.902048    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-envoy" pod "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" (UID: "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:40:44.903197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.903171    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-podinfo" pod "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" (UID: "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:40:44.904927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.904902    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-kube-api-access-m2dnl" pod "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" (UID: "2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920"). InnerVolumeSpecName "kube-api-access-m2dnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:40:44.992191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.992150    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:44.992191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.992190    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:44.992424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.992203    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:44.992424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.992216    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:44.992424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.992229    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:44.992424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.992243    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:44.992424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.992256    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:44.992424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.992270    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2dnl\" (UniqueName: \"kubernetes.io/projected/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-kube-api-access-m2dnl\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:44.992424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:44.992285    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:40:45.654371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:45.654347    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"
Nov 02 14:40:45.860003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:45.654379    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv" event={"ID":"2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920","Type":"ContainerDied","Data":"c9e647bdc0a2526221bf7fd9526fa62b8d802ac10226ec7f2ae602bcb97b0883"}
Nov 02 14:40:45.860003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:45.654439    5034 scope.go:122] "RemoveContainer" containerID="eb05fabc6444e265685056cf0dad1fd2bf1db0a8c63c636efccdf73b112371ae"
Nov 02 14:40:45.860003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:45.715548    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"]
Nov 02 14:40:45.860003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:45.728551    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8rxhv"]
Nov 02 14:40:46.665922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:46.665889    5034 generic.go:356] "Generic (PLEG): container finished" podID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerID="c69652c3bf2220dcb0d0e57aeac6982ddada3f044746d82c0efbb0dfc3eb4ad2" exitCode=0
Nov 02 14:40:46.666309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:46.665960    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" event={"ID":"e4e4ace3-b064-47cc-a5f0-031dac8babf2","Type":"ContainerDied","Data":"c69652c3bf2220dcb0d0e57aeac6982ddada3f044746d82c0efbb0dfc3eb4ad2"}
Nov 02 14:40:46.666392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:46.666375    5034 scope.go:122] "RemoveContainer" containerID="c69652c3bf2220dcb0d0e57aeac6982ddada3f044746d82c0efbb0dfc3eb4ad2"
Nov 02 14:40:46.668433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:46.668401    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:40:46.668532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:46.668479    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:40:46.668532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:46.668513    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:40:47.442849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:47.436963    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" path="/var/lib/kubelet/pods/2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920/volumes"
Nov 02 14:40:47.711310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:47.711224    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" event={"ID":"e4e4ace3-b064-47cc-a5f0-031dac8babf2","Type":"ContainerStarted","Data":"f1c075b8c6db1b7d1f153eeca21d5ba435e0feda4992bfc0c38e0af1573cd05d"}
Nov 02 14:40:47.943756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:47.943725    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:40:48.429914 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:40:48.429875    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:40:48.468213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:48.468187    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:40:48.484799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:48.484375    5034 patch_prober.go:29] interesting pod/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 14:40:48.484799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:48.484435    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 14:40:49.482812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:49.474940    5034 patch_prober.go:29] interesting pod/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 14:40:49.482812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:49.474999    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 14:40:50.468568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:50.468534    5034 patch_prober.go:29] interesting pod/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 14:40:50.468827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:50.468592    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 14:40:51.468636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:51.468606    5034 patch_prober.go:29] interesting pod/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 14:40:51.469190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:51.468668    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 14:40:52.468055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:52.468021    5034 patch_prober.go:29] interesting pod/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused" start-of-body=
Nov 02 14:40:52.468427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:52.468384    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.106:15021/healthz/ready\": dial tcp 10.132.1.106:15021: connect: connection refused"
Nov 02 14:40:53.041894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:53.037799    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"]
Nov 02 14:40:53.050347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:53.050320    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a02d9552-rtmbr"]
Nov 02 14:40:53.425292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:53.425205    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e7aed91-0172-4904-bcf1-2ec1dd897a6d" path="/var/lib/kubelet/pods/1e7aed91-0172-4904-bcf1-2ec1dd897a6d/volumes"
Nov 02 14:40:53.474427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:53.474400    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:40:53.475676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:40:53.475651    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:41:04.033621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:41:04.033587    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"]
Nov 02 14:41:04.046554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:41:04.046527    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d8d359-6g6lm"]
Nov 02 14:41:05.401814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:41:05.401758    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8028d5e-350d-4c29-8634-15422df32bb9" path="/var/lib/kubelet/pods/d8028d5e-350d-4c29-8634-15422df32bb9/volumes"
Nov 02 14:41:18.418556 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:41:18.418442    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:41:48.429997 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:41:48.429956    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:41:51.054846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:41:51.054802    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"]
Nov 02 14:41:51.068374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:41:51.068342    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e51146-25jng"]
Nov 02 14:41:51.403326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:41:51.403250    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6f7698-a0ef-4015-84b4-67b7f32c9641" path="/var/lib/kubelet/pods/6b6f7698-a0ef-4015-84b4-67b7f32c9641/volumes"
Nov 02 14:42:02.078120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:02.078046    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"]
Nov 02 14:42:02.084458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:02.084423    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa0e23a7-f5qq7"]
Nov 02 14:42:03.399355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:03.399321    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36e69ff-4bb2-4dd2-9009-11decc773623" path="/var/lib/kubelet/pods/f36e69ff-4bb2-4dd2-9009-11decc773623/volumes"
Nov 02 14:42:14.034414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:14.034376    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"]
Nov 02 14:42:14.047618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:14.047577    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ff1f80-cxm99"]
Nov 02 14:42:15.405528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:15.405484    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3845c16-0135-41b6-bc87-953357282374" path="/var/lib/kubelet/pods/c3845c16-0135-41b6-bc87-953357282374/volumes"
Nov 02 14:42:18.416016 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:42:18.415972    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:42:25.086277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:25.086239    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"]
Nov 02 14:42:25.100381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:25.100353    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa5047d2-whqcz"]
Nov 02 14:42:25.401568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:25.401476    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0873c506-22e6-4c25-a9a2-492f47a11c70" path="/var/lib/kubelet/pods/0873c506-22e6-4c25-a9a2-492f47a11c70/volumes"
Nov 02 14:42:33.039749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:33.039715    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"]
Nov 02 14:42:33.054203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:33.054176    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f33e918-dq555"]
Nov 02 14:42:33.398633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:33.398553    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37ad3ea2-1672-4f76-a3ad-01ec506622ea" path="/var/lib/kubelet/pods/37ad3ea2-1672-4f76-a3ad-01ec506622ea/volumes"
Nov 02 14:42:38.239764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.239731    5034 scope.go:122] "RemoveContainer" containerID="d3253f0cc1cbdb380064f88f536d591a08b7d2ad4665fac8bad54dbf6661f384"
Nov 02 14:42:38.251893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.251868    5034 scope.go:122] "RemoveContainer" containerID="7338b7bf00921c1ed5f939cda82a59578778cc8e976efd9457d44bc31265fb3b"
Nov 02 14:42:38.721099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.721013    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"]
Nov 02 14:42:38.722015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.721996    5034 cpu_manager.go:414] "containerMap: removing container" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy"
Nov 02 14:42:38.722077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.722017    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy"
Nov 02 14:42:38.722077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.722046    5034 cpu_manager.go:414] "containerMap: removing container" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy"
Nov 02 14:42:38.722077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.722053    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy"
Nov 02 14:42:38.722190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.722077    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy"
Nov 02 14:42:38.722190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.722083    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy"
Nov 02 14:42:38.722259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.722224    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2833a5ee-1c7b-4bbb-9a5b-e1ea68e8d920" containerName="istio-proxy"
Nov 02 14:42:38.722259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.722240    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy"
Nov 02 14:42:38.727979 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.727958    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.750300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.750273    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"]
Nov 02 14:42:38.758885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.758862    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-80133f-openshift-default-dockercfg-fn4tq\""
Nov 02 14:42:38.838611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.838575    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.838827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.838629    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqjc\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-kube-api-access-9sqjc\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.838920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.838837    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istiod-ca-cert\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.838974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.838945    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-data\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.839118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.839099    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-certs\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.839185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.839157    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-podinfo\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.839298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.839279    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-token\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.839364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.839314    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-credential-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.839529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.839505    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-envoy\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.941459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941431    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istiod-ca-cert\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.941608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941466    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-data\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.941608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941541    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-certs\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.941608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941573    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-podinfo\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.941739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941653    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-token\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.941739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941688    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-credential-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.941890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941819    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-envoy\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.941955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941891    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.941955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941939    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqjc\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-kube-api-access-9sqjc\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.942073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.941962    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-data\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.942073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.942002    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-certs\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.942301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.942278    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istiod-ca-cert\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.942397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.942352    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-credential-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.942464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.942430    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-socket\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.944235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.944216    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-envoy\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.944395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.944371    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-podinfo\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.969368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.969342    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-token\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:38.969748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:38.969723    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqjc\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-kube-api-access-9sqjc\") pod \"e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") " pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:39.043458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:39.043435    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:39.389653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:39.389622    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"]
Nov 02 14:42:39.390080 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:42:39.389761    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70178c0a_4f8a_49ab_8cb5_d565c3d3b9d3.slice/crio-2bfbde355284e0690b4dcadb1c7c7968965324ea6c5a05eac2c48bdfb12c0f95 WatchSource:0}: Error finding container 2bfbde355284e0690b4dcadb1c7c7968965324ea6c5a05eac2c48bdfb12c0f95: Status 404 returned error can't find the container with id 2bfbde355284e0690b4dcadb1c7c7968965324ea6c5a05eac2c48bdfb12c0f95
Nov 02 14:42:39.392988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:39.392953    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:42:39.393109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:39.393030    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:42:39.393109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:39.393077    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:42:40.161011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:40.160972    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" event={"ID":"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3","Type":"ContainerStarted","Data":"83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a"}
Nov 02 14:42:40.161196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:40.161030    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" event={"ID":"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3","Type":"ContainerStarted","Data":"2bfbde355284e0690b4dcadb1c7c7968965324ea6c5a05eac2c48bdfb12c0f95"}
Nov 02 14:42:40.240021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:40.239968    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" podStartSLOduration=2.239951181 podStartE2EDuration="2.239951181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:42:40.223814214 +0000 UTC m=+3550.975118643" watchObservedRunningTime="2026-11-02 14:42:40.239951181 +0000 UTC m=+3550.991255619"
Nov 02 14:42:41.043702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:41.043666    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:41.045752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:41.045726    5034 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 14:42:41.045874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:41.045795    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 14:42:42.043756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:42.043718    5034 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 14:42:42.044217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:42.043774    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 14:42:43.044193 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:43.044156    5034 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 14:42:43.044588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:43.044228    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 14:42:44.044503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:44.044465    5034 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 14:42:44.044945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:44.044519    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 14:42:44.048568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:44.048540    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"]
Nov 02 14:42:44.062260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:44.062236    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e55597c1-hwwql"]
Nov 02 14:42:45.043992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:45.043956    5034 patch_prober.go:29] interesting pod/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused" start-of-body=
Nov 02 14:42:45.044179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:45.044007    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.111:15021/healthz/ready\": dial tcp 10.132.1.111:15021: connect: connection refused"
Nov 02 14:42:45.424072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:45.423988    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b13f85-9472-40df-acfb-9fb13fb8b413" path="/var/lib/kubelet/pods/86b13f85-9472-40df-acfb-9fb13fb8b413/volumes"
Nov 02 14:42:46.050363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:46.050334    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:46.050702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:46.050672    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:46.051246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:42:46.051225    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:42:48.416338 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:42:48.416299    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:43:18.431641 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:43:18.431598    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:43:30.836843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:30.836814    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:43:30.841096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:30.841070    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:43:38.299232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:38.299195    5034 scope.go:122] "RemoveContainer" containerID="492c393493a34c6540a93a3ded939333ec50f146a130aafc7b5ed626ebc7db29"
Nov 02 14:43:38.318729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:38.318705    5034 scope.go:122] "RemoveContainer" containerID="f7b7cbef88e23becb89818f2667a4da039052a0e308d8272ca99b5a0b59725fe"
Nov 02 14:43:38.329006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:38.328981    5034 scope.go:122] "RemoveContainer" containerID="5313c7d3eca089f0ae331f3f3212ac103cddd7ca5a406e78732824508d4da21b"
Nov 02 14:43:38.341170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:38.341144    5034 scope.go:122] "RemoveContainer" containerID="8b171b23b10ccc5c770472289b371dd918b88d9d255147f1bec7fb81ab85f9be"
Nov 02 14:43:38.353819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:38.353799    5034 scope.go:122] "RemoveContainer" containerID="677f377d8da08749fdeef06fcd2874dbefeb5c69345dfff1da38922cb8f59867"
Nov 02 14:43:43.893902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.893861    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"]
Nov 02 14:43:43.895074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.895055    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cc2e99-f558-4e2c-be3a-ed110bebf2e9" containerName="istio-proxy"
Nov 02 14:43:43.901222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.901202    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:43.924068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.924043    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:43:43.929370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.929346    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"]
Nov 02 14:43:43.930531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.930502    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:43:43.930531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.930508    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:43:43.930703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.930510    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:43:43.967806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.967750    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d99bc3-h2skq\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:43.968005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.967986    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxr25\" (UniqueName: \"kubernetes.io/projected/af17e386-2606-46b4-896a-4d4ddaa3656b-kube-api-access-mxr25\") pod \"maas-api-revoke-sub-82d99bc3-h2skq\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:43.968243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:43.968213    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d99bc3-h2skq\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:44.069910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.069875    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d99bc3-h2skq\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:44.070110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.070024    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d99bc3-h2skq\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:44.070200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.070143    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxr25\" (UniqueName: \"kubernetes.io/projected/af17e386-2606-46b4-896a-4d4ddaa3656b-kube-api-access-mxr25\") pod \"maas-api-revoke-sub-82d99bc3-h2skq\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:44.070969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.070932    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d99bc3-h2skq\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:44.073196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.073156    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d99bc3-h2skq\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:44.110725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.110692    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxr25\" (UniqueName: \"kubernetes.io/projected/af17e386-2606-46b4-896a-4d4ddaa3656b-kube-api-access-mxr25\") pod \"maas-api-revoke-sub-82d99bc3-h2skq\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:44.219818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.219711    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:44.436667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.436605    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"]
Nov 02 14:43:44.444691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.444668    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"]
Nov 02 14:43:44.444899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.444855    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.501885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.501851    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f95719f-m8v4q\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.502086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.501925    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrs9x\" (UniqueName: \"kubernetes.io/projected/0524cef3-0ff9-49c1-b317-4cfd9d868a57-kube-api-access-hrs9x\") pod \"maas-api-revoke-sub-5f95719f-m8v4q\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.502356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.502330    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f95719f-m8v4q\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.553232 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:43:44.553192    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf17e386_2606_46b4_896a_4d4ddaa3656b.slice/crio-37a3b96509a17cfd8de9b609cd169215e867eebdc7a670414952820e2de4af30 WatchSource:0}: Error finding container 37a3b96509a17cfd8de9b609cd169215e867eebdc7a670414952820e2de4af30: Status 404 returned error can't find the container with id 37a3b96509a17cfd8de9b609cd169215e867eebdc7a670414952820e2de4af30
Nov 02 14:43:44.575037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.574998    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"]
Nov 02 14:43:44.603748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.603715    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f95719f-m8v4q\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.603872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.603835    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f95719f-m8v4q\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.603938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.603885    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrs9x\" (UniqueName: \"kubernetes.io/projected/0524cef3-0ff9-49c1-b317-4cfd9d868a57-kube-api-access-hrs9x\") pod \"maas-api-revoke-sub-5f95719f-m8v4q\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.604496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.604463    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f95719f-m8v4q\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.606127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.606106    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f95719f-m8v4q\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.632030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.632002    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrs9x\" (UniqueName: \"kubernetes.io/projected/0524cef3-0ff9-49c1-b317-4cfd9d868a57-kube-api-access-hrs9x\") pod \"maas-api-revoke-sub-5f95719f-m8v4q\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.772099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.772021    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:44.925454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:44.925413    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq" event={"ID":"af17e386-2606-46b4-896a-4d4ddaa3656b","Type":"ContainerStarted","Data":"37a3b96509a17cfd8de9b609cd169215e867eebdc7a670414952820e2de4af30"}
Nov 02 14:43:45.096046 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:43:45.095859    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0524cef3_0ff9_49c1_b317_4cfd9d868a57.slice/crio-dd32212edb031f6730ace7b3a255b9cc93f09106b22b8d7f543519d54b8b0329 WatchSource:0}: Error finding container dd32212edb031f6730ace7b3a255b9cc93f09106b22b8d7f543519d54b8b0329: Status 404 returned error can't find the container with id dd32212edb031f6730ace7b3a255b9cc93f09106b22b8d7f543519d54b8b0329
Nov 02 14:43:45.096728 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:45.096698    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"]
Nov 02 14:43:45.947846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:45.945664    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q" event={"ID":"0524cef3-0ff9-49c1-b317-4cfd9d868a57","Type":"ContainerStarted","Data":"5e9cc74d55c57842a626092192d85e79dfa711db99e81d8bfc817ae4a83e49f9"}
Nov 02 14:43:45.947846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:45.945735    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q" event={"ID":"0524cef3-0ff9-49c1-b317-4cfd9d868a57","Type":"ContainerStarted","Data":"dd32212edb031f6730ace7b3a255b9cc93f09106b22b8d7f543519d54b8b0329"}
Nov 02 14:43:45.949466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:45.949436    5034 generic.go:356] "Generic (PLEG): container finished" podID="af17e386-2606-46b4-896a-4d4ddaa3656b" containerID="903c3abce5f4413f9244fac825fb2471b1a8487c69dc54b277ea7ffb90b24ab5" exitCode=0
Nov 02 14:43:45.949614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:45.949550    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq" event={"ID":"af17e386-2606-46b4-896a-4d4ddaa3656b","Type":"ContainerDied","Data":"903c3abce5f4413f9244fac825fb2471b1a8487c69dc54b277ea7ffb90b24ab5"}
Nov 02 14:43:45.998026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:45.997957    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q" podStartSLOduration=1.997937346 podStartE2EDuration="1.997937346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:43:45.990127748 +0000 UTC m=+3616.741432188" watchObservedRunningTime="2026-11-02 14:43:45.997937346 +0000 UTC m=+3616.749241784"
Nov 02 14:43:46.961601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:46.961566    5034 generic.go:356] "Generic (PLEG): container finished" podID="0524cef3-0ff9-49c1-b317-4cfd9d868a57" containerID="5e9cc74d55c57842a626092192d85e79dfa711db99e81d8bfc817ae4a83e49f9" exitCode=0
Nov 02 14:43:46.962027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:46.961632    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q" event={"ID":"0524cef3-0ff9-49c1-b317-4cfd9d868a57","Type":"ContainerDied","Data":"5e9cc74d55c57842a626092192d85e79dfa711db99e81d8bfc817ae4a83e49f9"}
Nov 02 14:43:47.172180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.172152    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:47.285490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.285450    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-openshift-ca\") pod \"af17e386-2606-46b4-896a-4d4ddaa3656b\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") "
Nov 02 14:43:47.285755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.285736    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-xks-ca\") pod \"af17e386-2606-46b4-896a-4d4ddaa3656b\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") "
Nov 02 14:43:47.285935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.285913    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-openshift-ca" pod "af17e386-2606-46b4-896a-4d4ddaa3656b" (UID: "af17e386-2606-46b4-896a-4d4ddaa3656b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:47.285935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.285928    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af17e386-2606-46b4-896a-4d4ddaa3656b-kube-api-access-mxr25\" (UniqueName: \"kubernetes.io/projected/af17e386-2606-46b4-896a-4d4ddaa3656b-kube-api-access-mxr25\") pod \"af17e386-2606-46b4-896a-4d4ddaa3656b\" (UID: \"af17e386-2606-46b4-896a-4d4ddaa3656b\") "
Nov 02 14:43:47.286555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.286532    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:47.287847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.287815    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-xks-ca" pod "af17e386-2606-46b4-896a-4d4ddaa3656b" (UID: "af17e386-2606-46b4-896a-4d4ddaa3656b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:43:47.287977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.287957    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af17e386-2606-46b4-896a-4d4ddaa3656b-kube-api-access-mxr25" pod "af17e386-2606-46b4-896a-4d4ddaa3656b" (UID: "af17e386-2606-46b4-896a-4d4ddaa3656b"). InnerVolumeSpecName "kube-api-access-mxr25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:47.387601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.387566    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af17e386-2606-46b4-896a-4d4ddaa3656b-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:47.387601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.387596    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxr25\" (UniqueName: \"kubernetes.io/projected/af17e386-2606-46b4-896a-4d4ddaa3656b-kube-api-access-mxr25\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:47.974844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.973472    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"
Nov 02 14:43:47.974844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.973504    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq" event={"ID":"af17e386-2606-46b4-896a-4d4ddaa3656b","Type":"ContainerDied","Data":"37a3b96509a17cfd8de9b609cd169215e867eebdc7a670414952820e2de4af30"}
Nov 02 14:43:47.974844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:47.973554    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37a3b96509a17cfd8de9b609cd169215e867eebdc7a670414952820e2de4af30"
Nov 02 14:43:48.178894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.178868    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:48.303477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.303449    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-openshift-ca\") pod \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") "
Nov 02 14:43:48.303654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.303584    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-xks-ca\") pod \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") "
Nov 02 14:43:48.303808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.303789    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0524cef3-0ff9-49c1-b317-4cfd9d868a57-kube-api-access-hrs9x\" (UniqueName: \"kubernetes.io/projected/0524cef3-0ff9-49c1-b317-4cfd9d868a57-kube-api-access-hrs9x\") pod \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\" (UID: \"0524cef3-0ff9-49c1-b317-4cfd9d868a57\") "
Nov 02 14:43:48.303893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.303867    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-openshift-ca" pod "0524cef3-0ff9-49c1-b317-4cfd9d868a57" (UID: "0524cef3-0ff9-49c1-b317-4cfd9d868a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:48.304503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.304478    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:48.306023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.305995    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0524cef3-0ff9-49c1-b317-4cfd9d868a57-kube-api-access-hrs9x" pod "0524cef3-0ff9-49c1-b317-4cfd9d868a57" (UID: "0524cef3-0ff9-49c1-b317-4cfd9d868a57"). InnerVolumeSpecName "kube-api-access-hrs9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:48.306368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.306341    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-xks-ca" pod "0524cef3-0ff9-49c1-b317-4cfd9d868a57" (UID: "0524cef3-0ff9-49c1-b317-4cfd9d868a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:43:48.406590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.406558    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrs9x\" (UniqueName: \"kubernetes.io/projected/0524cef3-0ff9-49c1-b317-4cfd9d868a57-kube-api-access-hrs9x\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:48.406590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.406588    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0524cef3-0ff9-49c1-b317-4cfd9d868a57-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:48.439831 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:43:48.437199    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:43:48.983936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.983902    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"
Nov 02 14:43:48.984353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.983910    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q" event={"ID":"0524cef3-0ff9-49c1-b317-4cfd9d868a57","Type":"ContainerDied","Data":"dd32212edb031f6730ace7b3a255b9cc93f09106b22b8d7f543519d54b8b0329"}
Nov 02 14:43:48.984353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:48.984018    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd32212edb031f6730ace7b3a255b9cc93f09106b22b8d7f543519d54b8b0329"
Nov 02 14:43:54.088843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:54.082976    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"]
Nov 02 14:43:54.088843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:54.083235    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy" containerID="cri-o://f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9" gracePeriod=30
Nov 02 14:43:55.458178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.458142    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"]
Nov 02 14:43:55.459131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.459111    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0524cef3-0ff9-49c1-b317-4cfd9d868a57" containerName="revoke-keys"
Nov 02 14:43:55.459197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.459133    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0524cef3-0ff9-49c1-b317-4cfd9d868a57" containerName="revoke-keys"
Nov 02 14:43:55.459197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.459156    5034 cpu_manager.go:414] "containerMap: removing container" podUID="af17e386-2606-46b4-896a-4d4ddaa3656b" containerName="revoke-keys"
Nov 02 14:43:55.459197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.459162    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af17e386-2606-46b4-896a-4d4ddaa3656b" containerName="revoke-keys"
Nov 02 14:43:55.459375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.459364    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0524cef3-0ff9-49c1-b317-4cfd9d868a57" containerName="revoke-keys"
Nov 02 14:43:55.459413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.459378    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="af17e386-2606-46b4-896a-4d4ddaa3656b" containerName="revoke-keys"
Nov 02 14:43:55.463883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.463863    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.483769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.483740    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"]
Nov 02 14:43:55.490185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.490150    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:43:55.490185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.490150    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:43:55.490185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.490182    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:43:55.490445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.490273    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:43:55.549600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.549568    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trfks\" (UniqueName: \"kubernetes.io/projected/21010870-cadd-4750-a4b4-c1be487a6424-kube-api-access-trfks\") pod \"maas-api-revoke-sub-7c5d9c1c-2pv99\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.549794 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.549637    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c5d9c1c-2pv99\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.549892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.549825    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c5d9c1c-2pv99\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.650876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.650845    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trfks\" (UniqueName: \"kubernetes.io/projected/21010870-cadd-4750-a4b4-c1be487a6424-kube-api-access-trfks\") pod \"maas-api-revoke-sub-7c5d9c1c-2pv99\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.651015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.650915    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c5d9c1c-2pv99\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.651062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.651050    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c5d9c1c-2pv99\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.651579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.651561    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c5d9c1c-2pv99\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.653484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.653458    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c5d9c1c-2pv99\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.675633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.675607    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trfks\" (UniqueName: \"kubernetes.io/projected/21010870-cadd-4750-a4b4-c1be487a6424-kube-api-access-trfks\") pod \"maas-api-revoke-sub-7c5d9c1c-2pv99\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:55.781138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:55.781108    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:56.092967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:56.092939    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"]
Nov 02 14:43:56.093177 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:43:56.093151    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21010870_cadd_4750_a4b4_c1be487a6424.slice/crio-38b3df21116528b6a83f6e2298bbd90db7b31e62edbfcc20f1434331b531c2cf WatchSource:0}: Error finding container 38b3df21116528b6a83f6e2298bbd90db7b31e62edbfcc20f1434331b531c2cf: Status 404 returned error can't find the container with id 38b3df21116528b6a83f6e2298bbd90db7b31e62edbfcc20f1434331b531c2cf
Nov 02 14:43:56.149483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:56.149440    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99" event={"ID":"21010870-cadd-4750-a4b4-c1be487a6424","Type":"ContainerStarted","Data":"38b3df21116528b6a83f6e2298bbd90db7b31e62edbfcc20f1434331b531c2cf"}
Nov 02 14:43:57.161906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:57.161873    5034 generic.go:356] "Generic (PLEG): container finished" podID="21010870-cadd-4750-a4b4-c1be487a6424" containerID="110bdc90c3ca03fae53c30c36e91c41a2f671655356ea0a4bd7b5f1600493d36" exitCode=0
Nov 02 14:43:57.162308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:57.161955    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99" event={"ID":"21010870-cadd-4750-a4b4-c1be487a6424","Type":"ContainerDied","Data":"110bdc90c3ca03fae53c30c36e91c41a2f671655356ea0a4bd7b5f1600493d36"}
Nov 02 14:43:57.282563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:57.282533    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:43:57.282714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:57.282591    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:43:58.445145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.445079    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:58.511668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.511626    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21010870-cadd-4750-a4b4-c1be487a6424-kube-api-access-trfks\" (UniqueName: \"kubernetes.io/projected/21010870-cadd-4750-a4b4-c1be487a6424-kube-api-access-trfks\") pod \"21010870-cadd-4750-a4b4-c1be487a6424\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") "
Nov 02 14:43:58.511885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.511864    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-xks-ca\") pod \"21010870-cadd-4750-a4b4-c1be487a6424\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") "
Nov 02 14:43:58.511978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.511927    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-openshift-ca\") pod \"21010870-cadd-4750-a4b4-c1be487a6424\" (UID: \"21010870-cadd-4750-a4b4-c1be487a6424\") "
Nov 02 14:43:58.514512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.513209    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-openshift-ca" pod "21010870-cadd-4750-a4b4-c1be487a6424" (UID: "21010870-cadd-4750-a4b4-c1be487a6424"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:58.517943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.517871    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21010870-cadd-4750-a4b4-c1be487a6424-kube-api-access-trfks" pod "21010870-cadd-4750-a4b4-c1be487a6424" (UID: "21010870-cadd-4750-a4b4-c1be487a6424"). InnerVolumeSpecName "kube-api-access-trfks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:58.519763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.519733    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-xks-ca" pod "21010870-cadd-4750-a4b4-c1be487a6424" (UID: "21010870-cadd-4750-a4b4-c1be487a6424"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:43:58.614565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.614535    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:58.614565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.614562    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21010870-cadd-4750-a4b4-c1be487a6424-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:58.614741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:58.614572    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trfks\" (UniqueName: \"kubernetes.io/projected/21010870-cadd-4750-a4b4-c1be487a6424-kube-api-access-trfks\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:59.183280 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.183249    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"
Nov 02 14:43:59.183470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.183271    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99" event={"ID":"21010870-cadd-4750-a4b4-c1be487a6424","Type":"ContainerDied","Data":"38b3df21116528b6a83f6e2298bbd90db7b31e62edbfcc20f1434331b531c2cf"}
Nov 02 14:43:59.183470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.183358    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38b3df21116528b6a83f6e2298bbd90db7b31e62edbfcc20f1434331b531c2cf"
Nov 02 14:43:59.386675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.386379    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:43:59.427896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.427868    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-data\") pod \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") "
Nov 02 14:43:59.428063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428009    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-certs\") pod \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") "
Nov 02 14:43:59.428063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428048    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-kube-api-access-pjhz5\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-kube-api-access-pjhz5\") pod \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") "
Nov 02 14:43:59.428192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428117    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-envoy\") pod \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") "
Nov 02 14:43:59.428252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428228    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-socket\") pod \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") "
Nov 02 14:43:59.428305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428259    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istiod-ca-cert\") pod \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") "
Nov 02 14:43:59.428357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428328    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-podinfo\") pod \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") "
Nov 02 14:43:59.428461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428374    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-token\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-token\") pod \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") "
Nov 02 14:43:59.428461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428455    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-credential-socket\") pod \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\" (UID: \"3c2a56dd-c87f-4e98-91c8-e628aac57f7b\") "
Nov 02 14:43:59.428559 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428467    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-certs" pod "3c2a56dd-c87f-4e98-91c8-e628aac57f7b" (UID: "3c2a56dd-c87f-4e98-91c8-e628aac57f7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:59.428726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428702    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-data" pod "3c2a56dd-c87f-4e98-91c8-e628aac57f7b" (UID: "3c2a56dd-c87f-4e98-91c8-e628aac57f7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:59.428974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.428895    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-credential-socket" pod "3c2a56dd-c87f-4e98-91c8-e628aac57f7b" (UID: "3c2a56dd-c87f-4e98-91c8-e628aac57f7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:59.429277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.429037    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istiod-ca-cert" pod "3c2a56dd-c87f-4e98-91c8-e628aac57f7b" (UID: "3c2a56dd-c87f-4e98-91c8-e628aac57f7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:43:59.429517 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.429471    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-socket" pod "3c2a56dd-c87f-4e98-91c8-e628aac57f7b" (UID: "3c2a56dd-c87f-4e98-91c8-e628aac57f7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:59.433332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.433311    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:59.433479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.433465    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:59.433589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.433577    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:59.433685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.433672    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:59.433767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.433755    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:59.433872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.433323    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-podinfo" pod "3c2a56dd-c87f-4e98-91c8-e628aac57f7b" (UID: "3c2a56dd-c87f-4e98-91c8-e628aac57f7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:43:59.433962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.433347    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-envoy" pod "3c2a56dd-c87f-4e98-91c8-e628aac57f7b" (UID: "3c2a56dd-c87f-4e98-91c8-e628aac57f7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:43:59.434043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.434021    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-kube-api-access-pjhz5" pod "3c2a56dd-c87f-4e98-91c8-e628aac57f7b" (UID: "3c2a56dd-c87f-4e98-91c8-e628aac57f7b"). InnerVolumeSpecName "kube-api-access-pjhz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:59.434559 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.434537    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-token" pod "3c2a56dd-c87f-4e98-91c8-e628aac57f7b" (UID: "3c2a56dd-c87f-4e98-91c8-e628aac57f7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:43:59.535469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.535433    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhz5\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-kube-api-access-pjhz5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:59.535469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.535466    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:59.535469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.535477    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:43:59.535937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:43:59.535486    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2a56dd-c87f-4e98-91c8-e628aac57f7b-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:00.194918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.194882    5034 generic.go:356] "Generic (PLEG): container finished" podID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerID="f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9" exitCode=0
Nov 02 14:44:00.195123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.194936    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"
Nov 02 14:44:00.195123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.194966    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" event={"ID":"3c2a56dd-c87f-4e98-91c8-e628aac57f7b","Type":"ContainerDied","Data":"f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9"}
Nov 02 14:44:00.195123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.195035    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm" event={"ID":"3c2a56dd-c87f-4e98-91c8-e628aac57f7b","Type":"ContainerDied","Data":"69afc11fe8a6f1223f6207ddb0e7facf8e734b55b6d95639052781cd34327122"}
Nov 02 14:44:00.195123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.195060    5034 scope.go:122] "RemoveContainer" containerID="f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9"
Nov 02 14:44:00.208948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.208835    5034 scope.go:122] "RemoveContainer" containerID="8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f"
Nov 02 14:44:00.220117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.220079    5034 scope.go:122] "RemoveContainer" containerID="f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9"
Nov 02 14:44:00.220370 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:44:00.220348    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9\": container with ID starting with f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9 not found: ID does not exist" containerID="f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9"
Nov 02 14:44:00.220443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.220384    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9"} err="failed to get container status \"f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9\": rpc error: code = NotFound desc = could not find container \"f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9\": container with ID starting with f282198518431c7706025805a0f0e33b9696770985b915540a398001a2ef8bb9 not found: ID does not exist"
Nov 02 14:44:00.220443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.220406    5034 scope.go:122] "RemoveContainer" containerID="8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f"
Nov 02 14:44:00.220645 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:44:00.220627    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f\": container with ID starting with 8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f not found: ID does not exist" containerID="8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f"
Nov 02 14:44:00.220702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.220650    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f"} err="failed to get container status \"8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f\": rpc error: code = NotFound desc = could not find container \"8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f\": container with ID starting with 8d719cdca99bec8fcabe4d2014d73e1b631e2292287021c99e713a999090b58f not found: ID does not exist"
Nov 02 14:44:00.257199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.257172    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"]
Nov 02 14:44:00.271645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:00.271619    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-2daae4-openshift-default-998d889bc-z6xxm"]
Nov 02 14:44:01.401494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:01.401457    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" path="/var/lib/kubelet/pods/3c2a56dd-c87f-4e98-91c8-e628aac57f7b/volumes"
Nov 02 14:44:05.189316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.189282    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"]
Nov 02 14:44:05.189766 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.189505    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7" podUID="721d9747-d92b-47fc-9f0d-66e9d7a47db8" containerName="discovery" containerID="cri-o://b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73" gracePeriod=30
Nov 02 14:44:05.265413 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:44:05.265374    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721d9747_d92b_47fc_9f0d_66e9d7a47db8.slice/crio-b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 14:44:05.580548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.578837    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:44:05.662743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.662706    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-ca-configmap\") pod \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") "
Nov 02 14:44:05.662944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.662865    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-cacerts\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-cacerts\") pod \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") "
Nov 02 14:44:05.663032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.662978    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-token\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-token\") pod \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") "
Nov 02 14:44:05.663143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.663123    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-dns-cert\") pod \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") "
Nov 02 14:44:05.663213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.663171    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-ca-configmap" pod "721d9747-d92b-47fc-9f0d-66e9d7a47db8" (UID: "721d9747-d92b-47fc-9f0d-66e9d7a47db8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:05.663278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.663215    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-kube-api-access-z4ldw\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-kube-api-access-z4ldw\") pod \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") "
Nov 02 14:44:05.663350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.663332    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721d9747-d92b-47fc-9f0d-66e9d7a47db8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/721d9747-d92b-47fc-9f0d-66e9d7a47db8-local-certs\") pod \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") "
Nov 02 14:44:05.663453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.663436    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-kubeconfig\") pod \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\" (UID: \"721d9747-d92b-47fc-9f0d-66e9d7a47db8\") "
Nov 02 14:44:05.664472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.664443    5034 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-ca-configmap\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:05.667930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.667902    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-cacerts" pod "721d9747-d92b-47fc-9f0d-66e9d7a47db8" (UID: "721d9747-d92b-47fc-9f0d-66e9d7a47db8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:05.668053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.667926    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-dns-cert" pod "721d9747-d92b-47fc-9f0d-66e9d7a47db8" (UID: "721d9747-d92b-47fc-9f0d-66e9d7a47db8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:05.669021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.668986    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-kubeconfig" pod "721d9747-d92b-47fc-9f0d-66e9d7a47db8" (UID: "721d9747-d92b-47fc-9f0d-66e9d7a47db8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:05.669420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.669394    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-kube-api-access-z4ldw" pod "721d9747-d92b-47fc-9f0d-66e9d7a47db8" (UID: "721d9747-d92b-47fc-9f0d-66e9d7a47db8"). InnerVolumeSpecName "kube-api-access-z4ldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:05.679258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.679215    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721d9747-d92b-47fc-9f0d-66e9d7a47db8-local-certs" pod "721d9747-d92b-47fc-9f0d-66e9d7a47db8" (UID: "721d9747-d92b-47fc-9f0d-66e9d7a47db8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:05.679403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.679376    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-token" pod "721d9747-d92b-47fc-9f0d-66e9d7a47db8" (UID: "721d9747-d92b-47fc-9f0d-66e9d7a47db8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:05.766458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.766421    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:05.766458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.766456    5034 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-csr-dns-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:05.766676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.766467    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4ldw\" (UniqueName: \"kubernetes.io/projected/721d9747-d92b-47fc-9f0d-66e9d7a47db8-kube-api-access-z4ldw\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:05.766676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.766477    5034 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/721d9747-d92b-47fc-9f0d-66e9d7a47db8-local-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:05.766676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.766486    5034 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-istio-kubeconfig\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:05.766676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:05.766494    5034 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/721d9747-d92b-47fc-9f0d-66e9d7a47db8-cacerts\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:06.272474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:06.272435    5034 generic.go:356] "Generic (PLEG): container finished" podID="721d9747-d92b-47fc-9f0d-66e9d7a47db8" containerID="b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73" exitCode=0
Nov 02 14:44:06.272975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:06.272504    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"
Nov 02 14:44:06.272975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:06.272513    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7" event={"ID":"721d9747-d92b-47fc-9f0d-66e9d7a47db8","Type":"ContainerDied","Data":"b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73"}
Nov 02 14:44:06.272975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:06.272584    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7" event={"ID":"721d9747-d92b-47fc-9f0d-66e9d7a47db8","Type":"ContainerDied","Data":"ea502c5ce74bac8e30be1c2e1f7d6c773126a586f1e33bec9322e88b11d19700"}
Nov 02 14:44:06.272975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:06.272608    5034 scope.go:122] "RemoveContainer" containerID="b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73"
Nov 02 14:44:06.288520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:06.288494    5034 scope.go:122] "RemoveContainer" containerID="b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73"
Nov 02 14:44:06.288939 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:44:06.288872    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73\": container with ID starting with b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73 not found: ID does not exist" containerID="b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73"
Nov 02 14:44:06.288939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:06.288910    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73"} err="failed to get container status \"b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73\": rpc error: code = NotFound desc = could not find container \"b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73\": container with ID starting with b7bb4c0ca47ad3973e2d2daf2ca83138274b1f15cce5133d8a8904f6f3617c73 not found: ID does not exist"
Nov 02 14:44:06.373911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:06.373833    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"]
Nov 02 14:44:06.382771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:06.382740    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-m2ls7"]
Nov 02 14:44:07.398824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:07.398792    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721d9747-d92b-47fc-9f0d-66e9d7a47db8" path="/var/lib/kubelet/pods/721d9747-d92b-47fc-9f0d-66e9d7a47db8/volumes"
Nov 02 14:44:10.609455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:10.609364    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"]
Nov 02 14:44:10.610185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:10.609623    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy" containerID="cri-o://c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca" gracePeriod=30
Nov 02 14:44:13.002368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:13.002334    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"]
Nov 02 14:44:13.002835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:13.002573    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy" containerID="cri-o://5eb0d731dd9ca14436163bf8c66641565a137566ccdc585e4591f1d8ab9eef7f" gracePeriod=30
Nov 02 14:44:15.423043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.422996    5034 patch_prober.go:29] interesting pod/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:44:15.423494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.423069    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:44:15.935327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.935298    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:44:15.945965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.945940    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-socket\") pod \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") "
Nov 02 14:44:15.946127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946005    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-data\") pod \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") "
Nov 02 14:44:15.946127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946074    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-token\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-token\") pod \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") "
Nov 02 14:44:15.946127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946104    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-kube-api-access-wk5ff\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-kube-api-access-wk5ff\") pod \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") "
Nov 02 14:44:15.946365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946238    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-socket" pod "bb27b4b1-3b93-415a-82b0-1a4cb112783c" (UID: "bb27b4b1-3b93-415a-82b0-1a4cb112783c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.946365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946286    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-credential-socket\") pod \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") "
Nov 02 14:44:15.946365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946341    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-envoy\") pod \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") "
Nov 02 14:44:15.946479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946397    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-data" pod "bb27b4b1-3b93-415a-82b0-1a4cb112783c" (UID: "bb27b4b1-3b93-415a-82b0-1a4cb112783c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.946479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946422    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-podinfo\") pod \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") "
Nov 02 14:44:15.946479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946462    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-certs\") pod \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") "
Nov 02 14:44:15.946631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946596    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-credential-socket" pod "bb27b4b1-3b93-415a-82b0-1a4cb112783c" (UID: "bb27b4b1-3b93-415a-82b0-1a4cb112783c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.946631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946601    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istiod-ca-cert\") pod \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\" (UID: \"bb27b4b1-3b93-415a-82b0-1a4cb112783c\") "
Nov 02 14:44:15.946894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946858    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-certs" pod "bb27b4b1-3b93-415a-82b0-1a4cb112783c" (UID: "bb27b4b1-3b93-415a-82b0-1a4cb112783c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:15.947080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.946925    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istiod-ca-cert" pod "bb27b4b1-3b93-415a-82b0-1a4cb112783c" (UID: "bb27b4b1-3b93-415a-82b0-1a4cb112783c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:15.947688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.947662    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.947815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.947694    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.947815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.947709    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.947815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.947723    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.947815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.947736    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:15.949726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.949696    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-podinfo" pod "bb27b4b1-3b93-415a-82b0-1a4cb112783c" (UID: "bb27b4b1-3b93-415a-82b0-1a4cb112783c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:44:15.949877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.949754    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-kube-api-access-wk5ff" pod "bb27b4b1-3b93-415a-82b0-1a4cb112783c" (UID: "bb27b4b1-3b93-415a-82b0-1a4cb112783c"). InnerVolumeSpecName "kube-api-access-wk5ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:15.950214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.950186    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-token" pod "bb27b4b1-3b93-415a-82b0-1a4cb112783c" (UID: "bb27b4b1-3b93-415a-82b0-1a4cb112783c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:15.950946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:15.950924    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-envoy" pod "bb27b4b1-3b93-415a-82b0-1a4cb112783c" (UID: "bb27b4b1-3b93-415a-82b0-1a4cb112783c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:16.049285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.049248    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.049285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.049279    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk5ff\" (UniqueName: \"kubernetes.io/projected/bb27b4b1-3b93-415a-82b0-1a4cb112783c-kube-api-access-wk5ff\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.049285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.049291    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.049649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.049301    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb27b4b1-3b93-415a-82b0-1a4cb112783c-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:16.376443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.376352    5034 generic.go:356] "Generic (PLEG): container finished" podID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerID="c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca" exitCode=0
Nov 02 14:44:16.376443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.376430    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"
Nov 02 14:44:16.376684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.376434    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" event={"ID":"bb27b4b1-3b93-415a-82b0-1a4cb112783c","Type":"ContainerDied","Data":"c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca"}
Nov 02 14:44:16.376684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.376580    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l" event={"ID":"bb27b4b1-3b93-415a-82b0-1a4cb112783c","Type":"ContainerDied","Data":"e58c8393465defb16855a70e26855d0defc509f87f68a13b7594d0ed430e75ed"}
Nov 02 14:44:16.376684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.376614    5034 scope.go:122] "RemoveContainer" containerID="c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca"
Nov 02 14:44:16.396712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.396694    5034 scope.go:122] "RemoveContainer" containerID="c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca"
Nov 02 14:44:16.397068 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:44:16.397046    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca\": container with ID starting with c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca not found: ID does not exist" containerID="c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca"
Nov 02 14:44:16.397154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.397082    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca"} err="failed to get container status \"c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca\": rpc error: code = NotFound desc = could not find container \"c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca\": container with ID starting with c237eaf2d23e3070f8994c012d850b30cda3c25670bc1bcf0652fa6991b784ca not found: ID does not exist"
Nov 02 14:44:16.440899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.440869    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"]
Nov 02 14:44:16.455621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:16.454766    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a02c35-openshift-default-5b6cb45989-cbf2l"]
Nov 02 14:44:17.468807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:17.468291    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" path="/var/lib/kubelet/pods/bb27b4b1-3b93-415a-82b0-1a4cb112783c/volumes"
Nov 02 14:44:18.196808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.195356    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"]
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197555    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197582    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197629    5034 cpu_manager.go:414] "containerMap: removing container" podUID="21010870-cadd-4750-a4b4-c1be487a6424" containerName="revoke-keys"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197638    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21010870-cadd-4750-a4b4-c1be487a6424" containerName="revoke-keys"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197658    5034 cpu_manager.go:414] "containerMap: removing container" podUID="721d9747-d92b-47fc-9f0d-66e9d7a47db8" containerName="discovery"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197666    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721d9747-d92b-47fc-9f0d-66e9d7a47db8" containerName="discovery"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197679    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197686    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197949    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb27b4b1-3b93-415a-82b0-1a4cb112783c" containerName="istio-proxy"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197971    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="21010870-cadd-4750-a4b4-c1be487a6424" containerName="revoke-keys"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197985    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="721d9747-d92b-47fc-9f0d-66e9d7a47db8" containerName="discovery"
Nov 02 14:44:18.201159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.197996    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy"
Nov 02 14:44:18.210647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.206050    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.225395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.224251    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"]
Nov 02 14:44:18.244826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.242508    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:44:18.244826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.242607    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:44:18.244826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.242635    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:44:18.244826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.242675    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:44:18.318721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.316752    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0610595-f9wrn\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.318721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.317154    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0610595-f9wrn\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.318721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.317233    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgz5\" (UniqueName: \"kubernetes.io/projected/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-kube-api-access-nmgz5\") pod \"maas-api-revoke-sub-a0610595-f9wrn\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.434533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.434457    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0610595-f9wrn\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.435655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.435621    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgz5\" (UniqueName: \"kubernetes.io/projected/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-kube-api-access-nmgz5\") pod \"maas-api-revoke-sub-a0610595-f9wrn\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.436223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.436201    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0610595-f9wrn\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.438446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.438423    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0610595-f9wrn\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.447634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.447606    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0610595-f9wrn\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.464547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.464393    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgz5\" (UniqueName: \"kubernetes.io/projected/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-kube-api-access-nmgz5\") pod \"maas-api-revoke-sub-a0610595-f9wrn\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.498801 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:44:18.498668    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:44:18.511877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.500181    5034 generic.go:356] "Generic (PLEG): container finished" podID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerID="5eb0d731dd9ca14436163bf8c66641565a137566ccdc585e4591f1d8ab9eef7f" exitCode=0
Nov 02 14:44:18.511877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.500220    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" event={"ID":"91aab7bc-5ef8-4037-a001-3fefd605f6b4","Type":"ContainerDied","Data":"5eb0d731dd9ca14436163bf8c66641565a137566ccdc585e4591f1d8ab9eef7f"}
Nov 02 14:44:18.511877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.500279    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr" event={"ID":"91aab7bc-5ef8-4037-a001-3fefd605f6b4","Type":"ContainerDied","Data":"afa0774e1077e07a32afded7fb2f762fedb37372fae829f2c7f24511e41112f7"}
Nov 02 14:44:18.511877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.500304    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afa0774e1077e07a32afded7fb2f762fedb37372fae829f2c7f24511e41112f7"
Nov 02 14:44:18.511877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.500323    5034 scope.go:122] "RemoveContainer" containerID="93a9c0ed4ac74e1f0bcc1dce77e1b322efa4cd805c9b9c7008a192eaacf59ae8"
Nov 02 14:44:18.511877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.508314    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 14:44:18.538315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.538284    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istiod-ca-cert\") pod \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") "
Nov 02 14:44:18.538471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.538416    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-envoy\") pod \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") "
Nov 02 14:44:18.538471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.538457    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-podinfo\") pod \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") "
Nov 02 14:44:18.538587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.538498    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-data\") pod \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") "
Nov 02 14:44:18.538646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.538608    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-kube-api-access-n7kr6\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-kube-api-access-n7kr6\") pod \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") "
Nov 02 14:44:18.538708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.538675    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-credential-socket\") pod \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") "
Nov 02 14:44:18.538874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.538853    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-token\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-token\") pod \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") "
Nov 02 14:44:18.539018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.538991    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-certs\") pod \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") "
Nov 02 14:44:18.539085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.539063    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-socket\") pod \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\" (UID: \"91aab7bc-5ef8-4037-a001-3fefd605f6b4\") "
Nov 02 14:44:18.539164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.539093    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-data" pod "91aab7bc-5ef8-4037-a001-3fefd605f6b4" (UID: "91aab7bc-5ef8-4037-a001-3fefd605f6b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:18.539164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.539113    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-credential-socket" pod "91aab7bc-5ef8-4037-a001-3fefd605f6b4" (UID: "91aab7bc-5ef8-4037-a001-3fefd605f6b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:18.539335 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.539310    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-certs" pod "91aab7bc-5ef8-4037-a001-3fefd605f6b4" (UID: "91aab7bc-5ef8-4037-a001-3fefd605f6b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:18.539451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.539429    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-socket" pod "91aab7bc-5ef8-4037-a001-3fefd605f6b4" (UID: "91aab7bc-5ef8-4037-a001-3fefd605f6b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:18.539736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.539682    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istiod-ca-cert" pod "91aab7bc-5ef8-4037-a001-3fefd605f6b4" (UID: "91aab7bc-5ef8-4037-a001-3fefd605f6b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:18.540200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.540178    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.540295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.540203    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.540295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.540218    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.540295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.540232    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.540295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.540249    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.543131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.543001    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-envoy" pod "91aab7bc-5ef8-4037-a001-3fefd605f6b4" (UID: "91aab7bc-5ef8-4037-a001-3fefd605f6b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:44:18.543131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.543001    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-podinfo" pod "91aab7bc-5ef8-4037-a001-3fefd605f6b4" (UID: "91aab7bc-5ef8-4037-a001-3fefd605f6b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:44:18.543131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.543059    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-kube-api-access-n7kr6" pod "91aab7bc-5ef8-4037-a001-3fefd605f6b4" (UID: "91aab7bc-5ef8-4037-a001-3fefd605f6b4"). InnerVolumeSpecName "kube-api-access-n7kr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:18.543131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.543092    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-token" pod "91aab7bc-5ef8-4037-a001-3fefd605f6b4" (UID: "91aab7bc-5ef8-4037-a001-3fefd605f6b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:18.575063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.574379    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:18.646830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.643033    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.646830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.643088    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.646830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.643104    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7kr6\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-kube-api-access-n7kr6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.646830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.643137    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91aab7bc-5ef8-4037-a001-3fefd605f6b4-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:18.940952 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:44:18.940918    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8f1d18_dcf9_42b1_9bd6_eec60d16da37.slice/crio-4babc9dc2cb27bfd4bc906e2e3cf723e1fcff17317b50cfa9d4f87b05f47b25c WatchSource:0}: Error finding container 4babc9dc2cb27bfd4bc906e2e3cf723e1fcff17317b50cfa9d4f87b05f47b25c: Status 404 returned error can't find the container with id 4babc9dc2cb27bfd4bc906e2e3cf723e1fcff17317b50cfa9d4f87b05f47b25c
Nov 02 14:44:18.946946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:18.946919    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"]
Nov 02 14:44:19.511941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:19.511905    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn" event={"ID":"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37","Type":"ContainerStarted","Data":"4babc9dc2cb27bfd4bc906e2e3cf723e1fcff17317b50cfa9d4f87b05f47b25c"}
Nov 02 14:44:19.513688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:19.513665    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"
Nov 02 14:44:19.673502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:19.673418    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"]
Nov 02 14:44:19.689838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:19.682082    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-92f249-openshift-default-5f64549dbc-wrltr"]
Nov 02 14:44:20.527264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:20.527227    5034 generic.go:356] "Generic (PLEG): container finished" podID="8f8f1d18-dcf9-42b1-9bd6-eec60d16da37" containerID="ab0ec1b1dada09de38895c8c7ae77838ed464180cc3967dd3cec057f6d525f86" exitCode=0
Nov 02 14:44:20.527667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:20.527289    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn" event={"ID":"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37","Type":"ContainerDied","Data":"ab0ec1b1dada09de38895c8c7ae77838ed464180cc3967dd3cec057f6d525f86"}
Nov 02 14:44:21.399759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.399728    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" path="/var/lib/kubelet/pods/91aab7bc-5ef8-4037-a001-3fefd605f6b4/volumes"
Nov 02 14:44:21.760934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.760908    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:21.868612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.868576    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-xks-ca\") pod \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") "
Nov 02 14:44:21.868810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.868644    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-kube-api-access-nmgz5\" (UniqueName: \"kubernetes.io/projected/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-kube-api-access-nmgz5\") pod \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") "
Nov 02 14:44:21.869007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.868982    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-openshift-ca\") pod \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\" (UID: \"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37\") "
Nov 02 14:44:21.869518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.869492    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-openshift-ca" pod "8f8f1d18-dcf9-42b1-9bd6-eec60d16da37" (UID: "8f8f1d18-dcf9-42b1-9bd6-eec60d16da37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:21.869904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.869883    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:21.876014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.875986    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-kube-api-access-nmgz5" pod "8f8f1d18-dcf9-42b1-9bd6-eec60d16da37" (UID: "8f8f1d18-dcf9-42b1-9bd6-eec60d16da37"). InnerVolumeSpecName "kube-api-access-nmgz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:21.890162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.890123    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-xks-ca" pod "8f8f1d18-dcf9-42b1-9bd6-eec60d16da37" (UID: "8f8f1d18-dcf9-42b1-9bd6-eec60d16da37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:44:21.972224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.971911    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:21.972224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:21.971948    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmgz5\" (UniqueName: \"kubernetes.io/projected/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37-kube-api-access-nmgz5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:22.547525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:22.547500    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"
Nov 02 14:44:22.547743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:22.547543    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn" event={"ID":"8f8f1d18-dcf9-42b1-9bd6-eec60d16da37","Type":"ContainerDied","Data":"4babc9dc2cb27bfd4bc906e2e3cf723e1fcff17317b50cfa9d4f87b05f47b25c"}
Nov 02 14:44:22.547743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:22.547592    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4babc9dc2cb27bfd4bc906e2e3cf723e1fcff17317b50cfa9d4f87b05f47b25c"
Nov 02 14:44:25.417322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:25.417285    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"]
Nov 02 14:44:25.437811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:25.437755    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"]
Nov 02 14:44:30.418186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:30.418114    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p" podUID="027cafe7-543b-49c1-80a8-8059071fd0b8" containerName="payload-processing" containerID="cri-o://6a94cc7c4dc6115070e97ff1637ac75ded26775a44ef71598223d4035aa1a052" gracePeriod=25
Nov 02 14:44:30.439007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:30.438971    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq" podUID="ead84791-daf7-42de-8b50-91cd4bab519d" containerName="payload-pre-processing" containerID="cri-o://397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33" gracePeriod=25
Nov 02 14:44:33.934577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:33.934536    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p" podUID="027cafe7-543b-49c1-80a8-8059071fd0b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:44:33.934577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:33.934564    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq" podUID="ead84791-daf7-42de-8b50-91cd4bab519d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:44:38.409436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:38.409402    5034 scope.go:122] "RemoveContainer" containerID="5eb0d731dd9ca14436163bf8c66641565a137566ccdc585e4591f1d8ab9eef7f"
Nov 02 14:44:38.440056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:38.439867    5034 scope.go:122] "RemoveContainer" containerID="825ab3f3678cbb697e85b1a676f6a281435694ff46b607a8b5715d8274cdf034"
Nov 02 14:44:43.934679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:43.934640    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p" podUID="027cafe7-543b-49c1-80a8-8059071fd0b8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:44:43.935137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:43.934651    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq" podUID="ead84791-daf7-42de-8b50-91cd4bab519d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:44:48.415353 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:44:48.415316    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:44:48.761371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:48.761333    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"]
Nov 02 14:44:50.803539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.803509    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:44:50.827395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.827361    5034 generic.go:356] "Generic (PLEG): container finished" podID="027cafe7-543b-49c1-80a8-8059071fd0b8" containerID="6a94cc7c4dc6115070e97ff1637ac75ded26775a44ef71598223d4035aa1a052" exitCode=0
Nov 02 14:44:50.827729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.827619    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p" event={"ID":"027cafe7-543b-49c1-80a8-8059071fd0b8","Type":"ContainerDied","Data":"6a94cc7c4dc6115070e97ff1637ac75ded26775a44ef71598223d4035aa1a052"}
Nov 02 14:44:50.829474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.829449    5034 generic.go:356] "Generic (PLEG): container finished" podID="ead84791-daf7-42de-8b50-91cd4bab519d" containerID="397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33" exitCode=0
Nov 02 14:44:50.829581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.829513    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"
Nov 02 14:44:50.829581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.829527    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq" event={"ID":"ead84791-daf7-42de-8b50-91cd4bab519d","Type":"ContainerDied","Data":"397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33"}
Nov 02 14:44:50.829581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.829566    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq" event={"ID":"ead84791-daf7-42de-8b50-91cd4bab519d","Type":"ContainerDied","Data":"edaf1be5a02233709442006c6ac9e9045436e3ab3eedc3116d21138f5a843471"}
Nov 02 14:44:50.829719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.829595    5034 scope.go:122] "RemoveContainer" containerID="397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33"
Nov 02 14:44:50.842232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.842214    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:44:50.883330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.883307    5034 scope.go:122] "RemoveContainer" containerID="397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33"
Nov 02 14:44:50.883809 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:44:50.883761    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33\": container with ID starting with 397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33 not found: ID does not exist" containerID="397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33"
Nov 02 14:44:50.883929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.883821    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33"} err="failed to get container status \"397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33\": rpc error: code = NotFound desc = could not find container \"397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33\": container with ID starting with 397ae209d296a7fd9e929a50690b2bf56211b030259192cead346218e80aec33 not found: ID does not exist"
Nov 02 14:44:50.891742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.891715    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead84791-daf7-42de-8b50-91cd4bab519d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead84791-daf7-42de-8b50-91cd4bab519d-plugins-config-volume\") pod \"ead84791-daf7-42de-8b50-91cd4bab519d\" (UID: \"ead84791-daf7-42de-8b50-91cd4bab519d\") "
Nov 02 14:44:50.891871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.891851    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead84791-daf7-42de-8b50-91cd4bab519d-kube-api-access-bvzqk\" (UniqueName: \"kubernetes.io/projected/ead84791-daf7-42de-8b50-91cd4bab519d-kube-api-access-bvzqk\") pod \"ead84791-daf7-42de-8b50-91cd4bab519d\" (UID: \"ead84791-daf7-42de-8b50-91cd4bab519d\") "
Nov 02 14:44:50.892091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.892069    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead84791-daf7-42de-8b50-91cd4bab519d-plugins-config-volume" pod "ead84791-daf7-42de-8b50-91cd4bab519d" (UID: "ead84791-daf7-42de-8b50-91cd4bab519d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:50.893295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.893266    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead84791-daf7-42de-8b50-91cd4bab519d-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:50.894210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.894188    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead84791-daf7-42de-8b50-91cd4bab519d-kube-api-access-bvzqk" pod "ead84791-daf7-42de-8b50-91cd4bab519d" (UID: "ead84791-daf7-42de-8b50-91cd4bab519d"). InnerVolumeSpecName "kube-api-access-bvzqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:50.994018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.993982    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027cafe7-543b-49c1-80a8-8059071fd0b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027cafe7-543b-49c1-80a8-8059071fd0b8-plugins-config-volume\") pod \"027cafe7-543b-49c1-80a8-8059071fd0b8\" (UID: \"027cafe7-543b-49c1-80a8-8059071fd0b8\") "
Nov 02 14:44:50.994242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.994225    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027cafe7-543b-49c1-80a8-8059071fd0b8-kube-api-access-2tx2t\" (UniqueName: \"kubernetes.io/projected/027cafe7-543b-49c1-80a8-8059071fd0b8-kube-api-access-2tx2t\") pod \"027cafe7-543b-49c1-80a8-8059071fd0b8\" (UID: \"027cafe7-543b-49c1-80a8-8059071fd0b8\") "
Nov 02 14:44:50.994365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.994337    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027cafe7-543b-49c1-80a8-8059071fd0b8-plugins-config-volume" pod "027cafe7-543b-49c1-80a8-8059071fd0b8" (UID: "027cafe7-543b-49c1-80a8-8059071fd0b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:44:50.994910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.994891    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvzqk\" (UniqueName: \"kubernetes.io/projected/ead84791-daf7-42de-8b50-91cd4bab519d-kube-api-access-bvzqk\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:50.994979 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.994915    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/027cafe7-543b-49c1-80a8-8059071fd0b8-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:50.996446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:50.996415    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027cafe7-543b-49c1-80a8-8059071fd0b8-kube-api-access-2tx2t" pod "027cafe7-543b-49c1-80a8-8059071fd0b8" (UID: "027cafe7-543b-49c1-80a8-8059071fd0b8"). InnerVolumeSpecName "kube-api-access-2tx2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:44:51.096465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:51.096390    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tx2t\" (UniqueName: \"kubernetes.io/projected/027cafe7-543b-49c1-80a8-8059071fd0b8-kube-api-access-2tx2t\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:44:51.184328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:51.184297    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"]
Nov 02 14:44:51.198103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:51.198071    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-09d1a5-6dbbccc475-8qtjq"]
Nov 02 14:44:51.400220 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:51.400143    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead84791-daf7-42de-8b50-91cd4bab519d" path="/var/lib/kubelet/pods/ead84791-daf7-42de-8b50-91cd4bab519d/volumes"
Nov 02 14:44:51.841486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:51.841456    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"
Nov 02 14:44:51.841976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:51.841483    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p" event={"ID":"027cafe7-543b-49c1-80a8-8059071fd0b8","Type":"ContainerDied","Data":"2ac9266e3eaa5001ddbb0ce1bffb8738d3e208fc2ffabb9d0d1ef2c118fdad5e"}
Nov 02 14:44:51.841976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:51.841551    5034 scope.go:122] "RemoveContainer" containerID="6a94cc7c4dc6115070e97ff1637ac75ded26775a44ef71598223d4035aa1a052"
Nov 02 14:44:51.901210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:51.901182    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"]
Nov 02 14:44:51.914767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:51.914740    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-09d1a5-6fc4cf8c85-dcc8p"]
Nov 02 14:44:53.399846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:53.399815    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027cafe7-543b-49c1-80a8-8059071fd0b8" path="/var/lib/kubelet/pods/027cafe7-543b-49c1-80a8-8059071fd0b8/volumes"
Nov 02 14:44:53.761857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:44:53.761821    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz" podUID="de00d3b1-f2d5-46c7-a29c-fb1576200196" containerName="payload-processing" containerID="cri-o://fd59f174a079fc1377c4ad159c94a7acb9899cea2e88cc5ddf51fa84d4b6b6f2" gracePeriod=25
Nov 02 14:45:00.149330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.149265    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"]
Nov 02 14:45:00.150501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150479    5034 cpu_manager.go:414] "containerMap: removing container" podUID="027cafe7-543b-49c1-80a8-8059071fd0b8" containerName="payload-processing"
Nov 02 14:45:00.150501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150502    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027cafe7-543b-49c1-80a8-8059071fd0b8" containerName="payload-processing"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150545    5034 cpu_manager.go:414] "containerMap: removing container" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150551    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150563    5034 cpu_manager.go:414] "containerMap: removing container" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150568    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150576    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150580    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150604    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ead84791-daf7-42de-8b50-91cd4bab519d" containerName="payload-pre-processing"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150609    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead84791-daf7-42de-8b50-91cd4bab519d" containerName="payload-pre-processing"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150620    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8f8f1d18-dcf9-42b1-9bd6-eec60d16da37" containerName="revoke-keys"
Nov 02 14:45:00.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150625    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8f1d18-dcf9-42b1-9bd6-eec60d16da37" containerName="revoke-keys"
Nov 02 14:45:00.151027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150771    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead84791-daf7-42de-8b50-91cd4bab519d" containerName="payload-pre-processing"
Nov 02 14:45:00.151027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150796    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8f1d18-dcf9-42b1-9bd6-eec60d16da37" containerName="revoke-keys"
Nov 02 14:45:00.151027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150809    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="027cafe7-543b-49c1-80a8-8059071fd0b8" containerName="payload-processing"
Nov 02 14:45:00.151027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150820    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy"
Nov 02 14:45:00.151027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150827    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="91aab7bc-5ef8-4037-a001-3fefd605f6b4" containerName="istio-proxy"
Nov 02 14:45:00.151027 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.150833    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2a56dd-c87f-4e98-91c8-e628aac57f7b" containerName="istio-proxy"
Nov 02 14:45:00.156588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.156567    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"]
Nov 02 14:45:00.156704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.156685    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:00.161631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.161611    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"]
Nov 02 14:45:00.161748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.161721    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:00.166665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.166641    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"]
Nov 02 14:45:00.166837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.166767    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:00.189172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.189145    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:45:00.193681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.192057    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:45:00.194337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.192177    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:45:00.215112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.214879    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"]
Nov 02 14:45:00.215880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.215858    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:00.233574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.233546    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"]
Nov 02 14:45:00.235139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.235116    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:00.236669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.236638    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5crt\" (UniqueName: \"kubernetes.io/projected/82af8d05-2c41-441f-b905-a1b186bfcd11-kube-api-access-l5crt\") pod \"maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh\" (UID: \"82af8d05-2c41-441f-b905-a1b186bfcd11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:00.236835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.236696    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de34363d-7a31-4d79-8471-9b4e69d181d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26\" (UID: \"de34363d-7a31-4d79-8471-9b4e69d181d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:00.236835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.236738    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77d21c-5627-41d7-ab81-ec7a4651f524-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp\" (UID: \"cd77d21c-5627-41d7-ab81-ec7a4651f524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:00.236835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.236765    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66kl2\" (UniqueName: \"kubernetes.io/projected/cd77d21c-5627-41d7-ab81-ec7a4651f524-kube-api-access-66kl2\") pod \"maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp\" (UID: \"cd77d21c-5627-41d7-ab81-ec7a4651f524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:00.237063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.236881    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82af8d05-2c41-441f-b905-a1b186bfcd11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh\" (UID: \"82af8d05-2c41-441f-b905-a1b186bfcd11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:00.237063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.236999    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac656679-7557-495a-a88e-8b166be2da0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt\" (UID: \"ac656679-7557-495a-a88e-8b166be2da0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:00.237063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.237047    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhb2\" (UniqueName: \"kubernetes.io/projected/de34363d-7a31-4d79-8471-9b4e69d181d5-kube-api-access-bvhb2\") pod \"maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26\" (UID: \"de34363d-7a31-4d79-8471-9b4e69d181d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:00.237225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.237183    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glhlx\" (UniqueName: \"kubernetes.io/projected/ac656679-7557-495a-a88e-8b166be2da0e-kube-api-access-glhlx\") pod \"maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt\" (UID: \"ac656679-7557-495a-a88e-8b166be2da0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:00.241920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.240208    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"]
Nov 02 14:45:00.246384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.246366    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:00.249154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.249013    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"]
Nov 02 14:45:00.249272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.249204    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:00.251660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.251634    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"]
Nov 02 14:45:00.260312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.260284    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"]
Nov 02 14:45:00.263550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.262296    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:00.266602 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.266247    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:00.270888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.270867    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"]
Nov 02 14:45:00.292324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.292296    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"]
Nov 02 14:45:00.292476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.292346    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"]
Nov 02 14:45:00.292476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.292438    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:00.301681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.301655    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"]
Nov 02 14:45:00.310806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.310479    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"]
Nov 02 14:45:00.338789 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.338747    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac656679-7557-495a-a88e-8b166be2da0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt\" (UID: \"ac656679-7557-495a-a88e-8b166be2da0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:00.338946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.338829    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhb2\" (UniqueName: \"kubernetes.io/projected/de34363d-7a31-4d79-8471-9b4e69d181d5-kube-api-access-bvhb2\") pod \"maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26\" (UID: \"de34363d-7a31-4d79-8471-9b4e69d181d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:00.338946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.338881    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcxk\" (UniqueName: \"kubernetes.io/projected/a29378f6-902d-4efb-977a-dd1f13dd5248-kube-api-access-fzcxk\") pod \"maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn\" (UID: \"a29378f6-902d-4efb-977a-dd1f13dd5248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:00.339084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.338951    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df67ed0b-ff8d-4774-a628-3acbf17eb2db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn\" (UID: \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:00.339084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339009    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glhlx\" (UniqueName: \"kubernetes.io/projected/ac656679-7557-495a-a88e-8b166be2da0e-kube-api-access-glhlx\") pod \"maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt\" (UID: \"ac656679-7557-495a-a88e-8b166be2da0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:00.339084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339042    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twplp\" (UniqueName: \"kubernetes.io/projected/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-kube-api-access-twplp\") pod \"maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk\" (UID: \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:00.339265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339075    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxwgg\" (UniqueName: \"kubernetes.io/projected/6d83cf56-9c22-47e5-aef4-087b779a2f65-kube-api-access-vxwgg\") pod \"maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr\" (UID: \"6d83cf56-9c22-47e5-aef4-087b779a2f65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:00.339265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339181    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvfb\" (UniqueName: \"kubernetes.io/projected/54f4abb8-f88b-42df-83c4-0dbffb278927-kube-api-access-bdvfb\") pod \"maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8\" (UID: \"54f4abb8-f88b-42df-83c4-0dbffb278927\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:00.339265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339235    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vc7z\" (UniqueName: \"kubernetes.io/projected/6518f1f5-1243-49f7-8bf2-b174142e754a-kube-api-access-9vc7z\") pod \"maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m\" (UID: \"6518f1f5-1243-49f7-8bf2-b174142e754a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:00.339438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339271    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29378f6-902d-4efb-977a-dd1f13dd5248-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn\" (UID: \"a29378f6-902d-4efb-977a-dd1f13dd5248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:00.339438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339306    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5crt\" (UniqueName: \"kubernetes.io/projected/82af8d05-2c41-441f-b905-a1b186bfcd11-kube-api-access-l5crt\") pod \"maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh\" (UID: \"82af8d05-2c41-441f-b905-a1b186bfcd11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:00.339438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339335    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de34363d-7a31-4d79-8471-9b4e69d181d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26\" (UID: \"de34363d-7a31-4d79-8471-9b4e69d181d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:00.339438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339367    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6518f1f5-1243-49f7-8bf2-b174142e754a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m\" (UID: \"6518f1f5-1243-49f7-8bf2-b174142e754a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:00.339438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339401    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77d21c-5627-41d7-ab81-ec7a4651f524-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp\" (UID: \"cd77d21c-5627-41d7-ab81-ec7a4651f524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:00.339438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339434    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66kl2\" (UniqueName: \"kubernetes.io/projected/cd77d21c-5627-41d7-ab81-ec7a4651f524-kube-api-access-66kl2\") pod \"maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp\" (UID: \"cd77d21c-5627-41d7-ab81-ec7a4651f524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:00.339831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339460    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqhp\" (UniqueName: \"kubernetes.io/projected/df67ed0b-ff8d-4774-a628-3acbf17eb2db-kube-api-access-hmqhp\") pod \"maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn\" (UID: \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:00.339831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339579    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83cf56-9c22-47e5-aef4-087b779a2f65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr\" (UID: \"6d83cf56-9c22-47e5-aef4-087b779a2f65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:00.339831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339613    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82af8d05-2c41-441f-b905-a1b186bfcd11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh\" (UID: \"82af8d05-2c41-441f-b905-a1b186bfcd11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:00.339831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339730    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk\" (UID: \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:00.339831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.339756    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4abb8-f88b-42df-83c4-0dbffb278927-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8\" (UID: \"54f4abb8-f88b-42df-83c4-0dbffb278927\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:00.341141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.341115    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac656679-7557-495a-a88e-8b166be2da0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt\" (UID: \"ac656679-7557-495a-a88e-8b166be2da0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:00.341377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.341354    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de34363d-7a31-4d79-8471-9b4e69d181d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26\" (UID: \"de34363d-7a31-4d79-8471-9b4e69d181d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:00.345595 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.341832    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82af8d05-2c41-441f-b905-a1b186bfcd11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh\" (UID: \"82af8d05-2c41-441f-b905-a1b186bfcd11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:00.345595 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.342367    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77d21c-5627-41d7-ab81-ec7a4651f524-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp\" (UID: \"cd77d21c-5627-41d7-ab81-ec7a4651f524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:00.346684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.346665    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"]
Nov 02 14:45:00.357154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.357134    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"]
Nov 02 14:45:00.362409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.361976    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"]
Nov 02 14:45:00.375526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.375497    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:00.404228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.404121    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"]
Nov 02 14:45:00.411538 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.411470    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66kl2\" (UniqueName: \"kubernetes.io/projected/cd77d21c-5627-41d7-ab81-ec7a4651f524-kube-api-access-66kl2\") pod \"maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp\" (UID: \"cd77d21c-5627-41d7-ab81-ec7a4651f524\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:00.413444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.413420    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:00.413581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.413557    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5crt\" (UniqueName: \"kubernetes.io/projected/82af8d05-2c41-441f-b905-a1b186bfcd11-kube-api-access-l5crt\") pod \"maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh\" (UID: \"82af8d05-2c41-441f-b905-a1b186bfcd11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:00.413852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.413832    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glhlx\" (UniqueName: \"kubernetes.io/projected/ac656679-7557-495a-a88e-8b166be2da0e-kube-api-access-glhlx\") pod \"maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt\" (UID: \"ac656679-7557-495a-a88e-8b166be2da0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:00.415672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.415647    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhb2\" (UniqueName: \"kubernetes.io/projected/de34363d-7a31-4d79-8471-9b4e69d181d5-kube-api-access-bvhb2\") pod \"maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26\" (UID: \"de34363d-7a31-4d79-8471-9b4e69d181d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:00.422675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.420825    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"]
Nov 02 14:45:00.428359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.428304    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"]
Nov 02 14:45:00.439513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.437065    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"]
Nov 02 14:45:00.450344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.449753    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83cf56-9c22-47e5-aef4-087b779a2f65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr\" (UID: \"6d83cf56-9c22-47e5-aef4-087b779a2f65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:00.450632 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.450608    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk\" (UID: \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:00.450849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.450750    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4abb8-f88b-42df-83c4-0dbffb278927-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8\" (UID: \"54f4abb8-f88b-42df-83c4-0dbffb278927\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:00.451057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.451040    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcxk\" (UniqueName: \"kubernetes.io/projected/a29378f6-902d-4efb-977a-dd1f13dd5248-kube-api-access-fzcxk\") pod \"maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn\" (UID: \"a29378f6-902d-4efb-977a-dd1f13dd5248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:00.451247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.451230    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df67ed0b-ff8d-4774-a628-3acbf17eb2db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn\" (UID: \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:00.451412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.451396    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twplp\" (UniqueName: \"kubernetes.io/projected/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-kube-api-access-twplp\") pod \"maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk\" (UID: \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:00.455346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.455288    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxwgg\" (UniqueName: \"kubernetes.io/projected/6d83cf56-9c22-47e5-aef4-087b779a2f65-kube-api-access-vxwgg\") pod \"maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr\" (UID: \"6d83cf56-9c22-47e5-aef4-087b779a2f65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:00.457112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.457073    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83cf56-9c22-47e5-aef4-087b779a2f65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr\" (UID: \"6d83cf56-9c22-47e5-aef4-087b779a2f65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:00.457246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.457125    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvfb\" (UniqueName: \"kubernetes.io/projected/54f4abb8-f88b-42df-83c4-0dbffb278927-kube-api-access-bdvfb\") pod \"maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8\" (UID: \"54f4abb8-f88b-42df-83c4-0dbffb278927\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:00.457246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.457213    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vc7z\" (UniqueName: \"kubernetes.io/projected/6518f1f5-1243-49f7-8bf2-b174142e754a-kube-api-access-9vc7z\") pod \"maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m\" (UID: \"6518f1f5-1243-49f7-8bf2-b174142e754a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:00.457455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.457260    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29378f6-902d-4efb-977a-dd1f13dd5248-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn\" (UID: \"a29378f6-902d-4efb-977a-dd1f13dd5248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:00.457455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.457322    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6518f1f5-1243-49f7-8bf2-b174142e754a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m\" (UID: \"6518f1f5-1243-49f7-8bf2-b174142e754a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:00.457455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.457377    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqhp\" (UniqueName: \"kubernetes.io/projected/df67ed0b-ff8d-4774-a628-3acbf17eb2db-kube-api-access-hmqhp\") pod \"maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn\" (UID: \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:00.457717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.457688    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk\" (UID: \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:00.458408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.458371    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4abb8-f88b-42df-83c4-0dbffb278927-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8\" (UID: \"54f4abb8-f88b-42df-83c4-0dbffb278927\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:00.458595 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.458580    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:00.461829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.459228    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df67ed0b-ff8d-4774-a628-3acbf17eb2db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn\" (UID: \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:00.461829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.460972    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29378f6-902d-4efb-977a-dd1f13dd5248-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn\" (UID: \"a29378f6-902d-4efb-977a-dd1f13dd5248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:00.461829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.461659    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6518f1f5-1243-49f7-8bf2-b174142e754a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m\" (UID: \"6518f1f5-1243-49f7-8bf2-b174142e754a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:00.477440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.477421    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"]
Nov 02 14:45:00.490044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.489768    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"]
Nov 02 14:45:00.497894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.497867    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:00.499864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.499748    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxwgg\" (UniqueName: \"kubernetes.io/projected/6d83cf56-9c22-47e5-aef4-087b779a2f65-kube-api-access-vxwgg\") pod \"maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr\" (UID: \"6d83cf56-9c22-47e5-aef4-087b779a2f65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:00.515455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.515375    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vc7z\" (UniqueName: \"kubernetes.io/projected/6518f1f5-1243-49f7-8bf2-b174142e754a-kube-api-access-9vc7z\") pod \"maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m\" (UID: \"6518f1f5-1243-49f7-8bf2-b174142e754a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:00.516967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.516071    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqhp\" (UniqueName: \"kubernetes.io/projected/df67ed0b-ff8d-4774-a628-3acbf17eb2db-kube-api-access-hmqhp\") pod \"maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn\" (UID: \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:00.517112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.517089    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:00.517764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.517734    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twplp\" (UniqueName: \"kubernetes.io/projected/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-kube-api-access-twplp\") pod \"maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk\" (UID: \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:00.519655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.519631    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:00.522497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.522449    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcxk\" (UniqueName: \"kubernetes.io/projected/a29378f6-902d-4efb-977a-dd1f13dd5248-kube-api-access-fzcxk\") pod \"maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn\" (UID: \"a29378f6-902d-4efb-977a-dd1f13dd5248\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:00.529449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.529420    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvfb\" (UniqueName: \"kubernetes.io/projected/54f4abb8-f88b-42df-83c4-0dbffb278927-kube-api-access-bdvfb\") pod \"maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8\" (UID: \"54f4abb8-f88b-42df-83c4-0dbffb278927\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:00.532428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.532270    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:00.533323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.533302    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:00.533412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.533318    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"]
Nov 02 14:45:00.533488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.533377    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"]
Nov 02 14:45:00.535729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.535674    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"]
Nov 02 14:45:00.542859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.542601    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"]
Nov 02 14:45:00.555184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.555003    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"]
Nov 02 14:45:00.564217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.562759    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bfda5e-d485-4a81-835b-4c369b3666e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq\" (UID: \"98bfda5e-d485-4a81-835b-4c369b3666e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:00.564217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.562944    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpqg\" (UniqueName: \"kubernetes.io/projected/98bfda5e-d485-4a81-835b-4c369b3666e4-kube-api-access-pgpqg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq\" (UID: \"98bfda5e-d485-4a81-835b-4c369b3666e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:00.564217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.563164    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n66lc\" (UniqueName: \"kubernetes.io/projected/5527c983-4d3e-4d15-8118-3f4f656b8051-kube-api-access-n66lc\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb\" (UID: \"5527c983-4d3e-4d15-8118-3f4f656b8051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:00.564217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.563200    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527c983-4d3e-4d15-8118-3f4f656b8051-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb\" (UID: \"5527c983-4d3e-4d15-8118-3f4f656b8051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:00.564217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.563323    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wjl4\" (UniqueName: \"kubernetes.io/projected/b407ad8b-5856-4f5f-baf7-baebc7fe9233-kube-api-access-5wjl4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w\" (UID: \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:00.564217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.563359    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b407ad8b-5856-4f5f-baf7-baebc7fe9233-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w\" (UID: \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:00.566594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.566571    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:00.568323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.568284    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"]
Nov 02 14:45:00.568843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.568824    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:00.575118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.574933    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:00.575630 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.575607    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"]
Nov 02 14:45:00.595582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.595560    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"]
Nov 02 14:45:00.597931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.597910    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:00.599652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.599632    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:00.608474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.608440    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"]
Nov 02 14:45:00.608667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.608652    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"]
Nov 02 14:45:00.608861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.608848    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"]
Nov 02 14:45:00.609086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.609072    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:00.614948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.614888    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"]
Nov 02 14:45:00.615234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.615186    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:00.642706 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.642683    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"]
Nov 02 14:45:00.643541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.643513    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:00.707701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.662875    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:00.710331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.709234    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:00.711484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.711448    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdbb368-93f7-4f30-89ef-881ec3528d6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn\" (UID: \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:00.711641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.711536    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wjl4\" (UniqueName: \"kubernetes.io/projected/b407ad8b-5856-4f5f-baf7-baebc7fe9233-kube-api-access-5wjl4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w\" (UID: \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:00.711641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.711572    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2\" (UID: \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:00.711641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.711627    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b407ad8b-5856-4f5f-baf7-baebc7fe9233-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w\" (UID: \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:00.716012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.715748    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b407ad8b-5856-4f5f-baf7-baebc7fe9233-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w\" (UID: \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:00.716012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.715854    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:00.718699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.714998    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9smp\" (UniqueName: \"kubernetes.io/projected/c63a0f7e-2b21-46b1-80d2-61d439daf047-kube-api-access-v9smp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x\" (UID: \"c63a0f7e-2b21-46b1-80d2-61d439daf047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.719891    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9989de-be08-4242-944b-9db03854c4ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv\" (UID: \"0e9989de-be08-4242-944b-9db03854c4ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720010    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwgsr\" (UniqueName: \"kubernetes.io/projected/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-kube-api-access-kwgsr\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s\" (UID: \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720062    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b296f5f-1030-4705-998d-4459d0706cb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr\" (UID: \"7b296f5f-1030-4705-998d-4459d0706cb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720117    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bfda5e-d485-4a81-835b-4c369b3666e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq\" (UID: \"98bfda5e-d485-4a81-835b-4c369b3666e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720196    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63a0f7e-2b21-46b1-80d2-61d439daf047-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x\" (UID: \"c63a0f7e-2b21-46b1-80d2-61d439daf047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720273    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx7sh\" (UniqueName: \"kubernetes.io/projected/5bdbb368-93f7-4f30-89ef-881ec3528d6c-kube-api-access-tx7sh\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn\" (UID: \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720325    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvv9\" (UniqueName: \"kubernetes.io/projected/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-kube-api-access-mmvv9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2\" (UID: \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720361    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpqg\" (UniqueName: \"kubernetes.io/projected/98bfda5e-d485-4a81-835b-4c369b3666e4-kube-api-access-pgpqg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq\" (UID: \"98bfda5e-d485-4a81-835b-4c369b3666e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720410    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkdk\" (UniqueName: \"kubernetes.io/projected/7b296f5f-1030-4705-998d-4459d0706cb0-kube-api-access-nrkdk\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr\" (UID: \"7b296f5f-1030-4705-998d-4459d0706cb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720440    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkpt2\" (UniqueName: \"kubernetes.io/projected/0e9989de-be08-4242-944b-9db03854c4ed-kube-api-access-hkpt2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv\" (UID: \"0e9989de-be08-4242-944b-9db03854c4ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720585    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s\" (UID: \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720685    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n66lc\" (UniqueName: \"kubernetes.io/projected/5527c983-4d3e-4d15-8118-3f4f656b8051-kube-api-access-n66lc\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb\" (UID: \"5527c983-4d3e-4d15-8118-3f4f656b8051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.720721    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527c983-4d3e-4d15-8118-3f4f656b8051-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb\" (UID: \"5527c983-4d3e-4d15-8118-3f4f656b8051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.721534    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527c983-4d3e-4d15-8118-3f4f656b8051-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb\" (UID: \"5527c983-4d3e-4d15-8118-3f4f656b8051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:00.722180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.721963    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bfda5e-d485-4a81-835b-4c369b3666e4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq\" (UID: \"98bfda5e-d485-4a81-835b-4c369b3666e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:00.759809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.728708    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"]
Nov 02 14:45:00.759809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.728755    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"]
Nov 02 14:45:00.759809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.732825    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:00.759809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.737468    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:00.759809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.745097    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wjl4\" (UniqueName: \"kubernetes.io/projected/b407ad8b-5856-4f5f-baf7-baebc7fe9233-kube-api-access-5wjl4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w\" (UID: \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:00.759809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.747271    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpqg\" (UniqueName: \"kubernetes.io/projected/98bfda5e-d485-4a81-835b-4c369b3666e4-kube-api-access-pgpqg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq\" (UID: \"98bfda5e-d485-4a81-835b-4c369b3666e4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:00.759809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.747359    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"]
Nov 02 14:45:00.782887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.766959    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"]
Nov 02 14:45:00.782887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.767002    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"]
Nov 02 14:45:00.782887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.768434    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n66lc\" (UniqueName: \"kubernetes.io/projected/5527c983-4d3e-4d15-8118-3f4f656b8051-kube-api-access-n66lc\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb\" (UID: \"5527c983-4d3e-4d15-8118-3f4f656b8051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:00.782887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.777184    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:00.792954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.790979    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:00.802374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.798877    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"]
Nov 02 14:45:00.818471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.818448    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.824478    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvv9\" (UniqueName: \"kubernetes.io/projected/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-kube-api-access-mmvv9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2\" (UID: \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.824546    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkdk\" (UniqueName: \"kubernetes.io/projected/7b296f5f-1030-4705-998d-4459d0706cb0-kube-api-access-nrkdk\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr\" (UID: \"7b296f5f-1030-4705-998d-4459d0706cb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.824578    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkpt2\" (UniqueName: \"kubernetes.io/projected/0e9989de-be08-4242-944b-9db03854c4ed-kube-api-access-hkpt2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv\" (UID: \"0e9989de-be08-4242-944b-9db03854c4ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.824700    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s\" (UID: \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.824867    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdbb368-93f7-4f30-89ef-881ec3528d6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn\" (UID: \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.824908    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2\" (UID: \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.824999    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9smp\" (UniqueName: \"kubernetes.io/projected/c63a0f7e-2b21-46b1-80d2-61d439daf047-kube-api-access-v9smp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x\" (UID: \"c63a0f7e-2b21-46b1-80d2-61d439daf047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.825030    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9989de-be08-4242-944b-9db03854c4ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv\" (UID: \"0e9989de-be08-4242-944b-9db03854c4ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.825090    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwgsr\" (UniqueName: \"kubernetes.io/projected/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-kube-api-access-kwgsr\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s\" (UID: \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.825130    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b296f5f-1030-4705-998d-4459d0706cb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr\" (UID: \"7b296f5f-1030-4705-998d-4459d0706cb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.825216    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63a0f7e-2b21-46b1-80d2-61d439daf047-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x\" (UID: \"c63a0f7e-2b21-46b1-80d2-61d439daf047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:00.826204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.825287    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx7sh\" (UniqueName: \"kubernetes.io/projected/5bdbb368-93f7-4f30-89ef-881ec3528d6c-kube-api-access-tx7sh\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn\" (UID: \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:00.844127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.834687    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s\" (UID: \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:00.844127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.835579    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2\" (UID: \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:00.844127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.835689    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"]
Nov 02 14:45:00.844127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.836111    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdbb368-93f7-4f30-89ef-881ec3528d6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn\" (UID: \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:00.844127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.837085    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63a0f7e-2b21-46b1-80d2-61d439daf047-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x\" (UID: \"c63a0f7e-2b21-46b1-80d2-61d439daf047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:00.844127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.837331    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"]
Nov 02 14:45:00.844127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.840888    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9989de-be08-4242-944b-9db03854c4ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv\" (UID: \"0e9989de-be08-4242-944b-9db03854c4ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:00.849079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.848859    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b296f5f-1030-4705-998d-4459d0706cb0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr\" (UID: \"7b296f5f-1030-4705-998d-4459d0706cb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:00.854341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.854262    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"]
Nov 02 14:45:00.868741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.868684    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkpt2\" (UniqueName: \"kubernetes.io/projected/0e9989de-be08-4242-944b-9db03854c4ed-kube-api-access-hkpt2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv\" (UID: \"0e9989de-be08-4242-944b-9db03854c4ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:00.868919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.868887    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwgsr\" (UniqueName: \"kubernetes.io/projected/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-kube-api-access-kwgsr\") pod \"maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s\" (UID: \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:00.883082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.881527    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:00.912625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.912558    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:00.928131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.926756    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj\" (UID: \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:00.928131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.926897    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dcb0f45-1ec6-43ac-a9af-571593753a63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn\" (UID: \"4dcb0f45-1ec6-43ac-a9af-571593753a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:00.928131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.926931    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708b38e0-5f4a-4669-84e7-f74b02a3425e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc\" (UID: \"708b38e0-5f4a-4669-84e7-f74b02a3425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:00.928131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.927002    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq9tx\" (UniqueName: \"kubernetes.io/projected/86a9e4f4-79f9-4e98-be5e-f871f11b183d-kube-api-access-mq9tx\") pod \"maas-api-key-cleanup-29893845-cgqjf\" (UID: \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:00.928131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.927209    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a9e4f4-79f9-4e98-be5e-f871f11b183d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893845-cgqjf\" (UID: \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:00.928131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.927257    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5v4q\" (UniqueName: \"kubernetes.io/projected/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-kube-api-access-m5v4q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj\" (UID: \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:00.928131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.927323    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgn6l\" (UniqueName: \"kubernetes.io/projected/4dcb0f45-1ec6-43ac-a9af-571593753a63-kube-api-access-xgn6l\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn\" (UID: \"4dcb0f45-1ec6-43ac-a9af-571593753a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:00.928131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.927424    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnkcn\" (UniqueName: \"kubernetes.io/projected/708b38e0-5f4a-4669-84e7-f74b02a3425e-kube-api-access-bnkcn\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc\" (UID: \"708b38e0-5f4a-4669-84e7-f74b02a3425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:00.962672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.962475    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkdk\" (UniqueName: \"kubernetes.io/projected/7b296f5f-1030-4705-998d-4459d0706cb0-kube-api-access-nrkdk\") pod \"maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr\" (UID: \"7b296f5f-1030-4705-998d-4459d0706cb0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:00.963752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.963632    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9smp\" (UniqueName: \"kubernetes.io/projected/c63a0f7e-2b21-46b1-80d2-61d439daf047-kube-api-access-v9smp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x\" (UID: \"c63a0f7e-2b21-46b1-80d2-61d439daf047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:00.968692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.964456    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvv9\" (UniqueName: \"kubernetes.io/projected/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-kube-api-access-mmvv9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2\" (UID: \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:00.968692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:00.965393    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx7sh\" (UniqueName: \"kubernetes.io/projected/5bdbb368-93f7-4f30-89ef-881ec3528d6c-kube-api-access-tx7sh\") pod \"maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn\" (UID: \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:01.031835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.031762    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj\" (UID: \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:01.031990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.031948    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dcb0f45-1ec6-43ac-a9af-571593753a63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn\" (UID: \"4dcb0f45-1ec6-43ac-a9af-571593753a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:01.032047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.031990    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708b38e0-5f4a-4669-84e7-f74b02a3425e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc\" (UID: \"708b38e0-5f4a-4669-84e7-f74b02a3425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:01.032104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.032076    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq9tx\" (UniqueName: \"kubernetes.io/projected/86a9e4f4-79f9-4e98-be5e-f871f11b183d-kube-api-access-mq9tx\") pod \"maas-api-key-cleanup-29893845-cgqjf\" (UID: \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:01.032343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.032322    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a9e4f4-79f9-4e98-be5e-f871f11b183d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893845-cgqjf\" (UID: \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:01.032424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.032384    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5v4q\" (UniqueName: \"kubernetes.io/projected/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-kube-api-access-m5v4q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj\" (UID: \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:01.032485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.032460    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgn6l\" (UniqueName: \"kubernetes.io/projected/4dcb0f45-1ec6-43ac-a9af-571593753a63-kube-api-access-xgn6l\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn\" (UID: \"4dcb0f45-1ec6-43ac-a9af-571593753a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:01.032539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.032509    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:01.032629 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.032606    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnkcn\" (UniqueName: \"kubernetes.io/projected/708b38e0-5f4a-4669-84e7-f74b02a3425e-kube-api-access-bnkcn\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc\" (UID: \"708b38e0-5f4a-4669-84e7-f74b02a3425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:01.033919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.033891    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708b38e0-5f4a-4669-84e7-f74b02a3425e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc\" (UID: \"708b38e0-5f4a-4669-84e7-f74b02a3425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:01.034390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.034364    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dcb0f45-1ec6-43ac-a9af-571593753a63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn\" (UID: \"4dcb0f45-1ec6-43ac-a9af-571593753a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:01.035170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.035124    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a9e4f4-79f9-4e98-be5e-f871f11b183d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893845-cgqjf\" (UID: \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:01.035273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.035174    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj\" (UID: \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:01.074621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.074172    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:01.121731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.120287    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5v4q\" (UniqueName: \"kubernetes.io/projected/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-kube-api-access-m5v4q\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj\" (UID: \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:01.122301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.122085    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:01.123599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.123561    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgn6l\" (UniqueName: \"kubernetes.io/projected/4dcb0f45-1ec6-43ac-a9af-571593753a63-kube-api-access-xgn6l\") pod \"maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn\" (UID: \"4dcb0f45-1ec6-43ac-a9af-571593753a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:01.134299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.128710    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnkcn\" (UniqueName: \"kubernetes.io/projected/708b38e0-5f4a-4669-84e7-f74b02a3425e-kube-api-access-bnkcn\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc\" (UID: \"708b38e0-5f4a-4669-84e7-f74b02a3425e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:01.135814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.134833    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq9tx\" (UniqueName: \"kubernetes.io/projected/86a9e4f4-79f9-4e98-be5e-f871f11b183d-kube-api-access-mq9tx\") pod \"maas-api-key-cleanup-29893845-cgqjf\" (UID: \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:01.148704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.147082    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:01.161197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.160439    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:01.180217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.176768    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:01.191037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.189673    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:01.204693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.204658    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:01.217868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.217612    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:01.249657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.249388    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:01.530736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.530707    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"]
Nov 02 14:45:01.594467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.594363    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"]
Nov 02 14:45:01.816729 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:01.816684    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82af8d05_2c41_441f_b905_a1b186bfcd11.slice/crio-93a6a5367393272affd047d1e16e0993c1678101d7ab59f6d27dcc2cb684f313 WatchSource:0}: Error finding container 93a6a5367393272affd047d1e16e0993c1678101d7ab59f6d27dcc2cb684f313: Status 404 returned error can't find the container with id 93a6a5367393272affd047d1e16e0993c1678101d7ab59f6d27dcc2cb684f313
Nov 02 14:45:01.825482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.825449    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"]
Nov 02 14:45:01.867492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.867464    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"]
Nov 02 14:45:01.871144 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:01.871110    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d83cf56_9c22_47e5_aef4_087b779a2f65.slice/crio-9a8e5b0bb2a149f5c4fc97cdbb81bab30505b4f98b6e8adac0180855e8ffc8fa WatchSource:0}: Error finding container 9a8e5b0bb2a149f5c4fc97cdbb81bab30505b4f98b6e8adac0180855e8ffc8fa: Status 404 returned error can't find the container with id 9a8e5b0bb2a149f5c4fc97cdbb81bab30505b4f98b6e8adac0180855e8ffc8fa
Nov 02 14:45:01.871489 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:01.871446    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde34363d_7a31_4d79_8471_9b4e69d181d5.slice/crio-265b91b3ffe0b1f10d48e2d420e511e41540ef1e5dcdd845148a254141c06445 WatchSource:0}: Error finding container 265b91b3ffe0b1f10d48e2d420e511e41540ef1e5dcdd845148a254141c06445: Status 404 returned error can't find the container with id 265b91b3ffe0b1f10d48e2d420e511e41540ef1e5dcdd845148a254141c06445
Nov 02 14:45:01.890268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:01.890238    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"]
Nov 02 14:45:02.053569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.053425    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" event={"ID":"ac656679-7557-495a-a88e-8b166be2da0e","Type":"ContainerStarted","Data":"3b67c93cafc6f5827b1c209749bf14d561176022e9ad87bd332ba972f02abdbd"}
Nov 02 14:45:02.055504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.055474    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh" event={"ID":"82af8d05-2c41-441f-b905-a1b186bfcd11","Type":"ContainerStarted","Data":"93a6a5367393272affd047d1e16e0993c1678101d7ab59f6d27dcc2cb684f313"}
Nov 02 14:45:02.057118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.057086    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" event={"ID":"cd77d21c-5627-41d7-ab81-ec7a4651f524","Type":"ContainerStarted","Data":"15dabd8cd07dbbeb9e4d5587ec943e743055ba1e4268ff96c04ac03617ebab35"}
Nov 02 14:45:02.059265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.059240    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" event={"ID":"6d83cf56-9c22-47e5-aef4-087b779a2f65","Type":"ContainerStarted","Data":"9a8e5b0bb2a149f5c4fc97cdbb81bab30505b4f98b6e8adac0180855e8ffc8fa"}
Nov 02 14:45:02.060846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.060824    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26" event={"ID":"de34363d-7a31-4d79-8471-9b4e69d181d5","Type":"ContainerStarted","Data":"265b91b3ffe0b1f10d48e2d420e511e41540ef1e5dcdd845148a254141c06445"}
Nov 02 14:45:02.105028 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.103818    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"]
Nov 02 14:45:02.111502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.111457    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"]
Nov 02 14:45:02.120506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.120383    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"]
Nov 02 14:45:02.125941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.125909    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"]
Nov 02 14:45:02.144712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.144686    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"]
Nov 02 14:45:02.154251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.154160    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"]
Nov 02 14:45:02.156076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.156056    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"]
Nov 02 14:45:02.187545 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.187505    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9989de_be08_4242_944b_9db03854c4ed.slice/crio-9996c239df406912ae540b85c3369a6326e34e364e46f8893638d05bc9c5a0d2 WatchSource:0}: Error finding container 9996c239df406912ae540b85c3369a6326e34e364e46f8893638d05bc9c5a0d2: Status 404 returned error can't find the container with id 9996c239df406912ae540b85c3369a6326e34e364e46f8893638d05bc9c5a0d2
Nov 02 14:45:02.193121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.193094    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"]
Nov 02 14:45:02.228543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.228513    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"]
Nov 02 14:45:02.240072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.240037    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"]
Nov 02 14:45:02.275750 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.275715    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3238cdd3_2e70_456b_afe8_6cdbf2b20b89.slice/crio-c152d54b2d26326414e5c3fdc0184af9efd6de905a7805fd18f804382776c68a WatchSource:0}: Error finding container c152d54b2d26326414e5c3fdc0184af9efd6de905a7805fd18f804382776c68a: Status 404 returned error can't find the container with id c152d54b2d26326414e5c3fdc0184af9efd6de905a7805fd18f804382776c68a
Nov 02 14:45:02.306298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.306219    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"]
Nov 02 14:45:02.319616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.319586    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"]
Nov 02 14:45:02.352615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.344965    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"]
Nov 02 14:45:02.389497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.389460    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"]
Nov 02 14:45:02.426906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.426505    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"]
Nov 02 14:45:02.433192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.433158    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"]
Nov 02 14:45:02.438545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.438515    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"]
Nov 02 14:45:02.474033 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.473999    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29378f6_902d_4efb_977a_dd1f13dd5248.slice/crio-5f6a4e95b24e551eeb840b70c8ce46f0ac57a6f55b2ddcbc0261643894e5f952 WatchSource:0}: Error finding container 5f6a4e95b24e551eeb840b70c8ce46f0ac57a6f55b2ddcbc0261643894e5f952: Status 404 returned error can't find the container with id 5f6a4e95b24e551eeb840b70c8ce46f0ac57a6f55b2ddcbc0261643894e5f952
Nov 02 14:45:02.477721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:02.477692    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"]
Nov 02 14:45:02.585245 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.585215    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bfda5e_d485_4a81_835b_4c369b3666e4.slice/crio-ca84e7847d9c5480dc8541dbaf2b9313e081c2dd049c6a980f211ca94bb133d7 WatchSource:0}: Error finding container ca84e7847d9c5480dc8541dbaf2b9313e081c2dd049c6a980f211ca94bb133d7: Status 404 returned error can't find the container with id ca84e7847d9c5480dc8541dbaf2b9313e081c2dd049c6a980f211ca94bb133d7
Nov 02 14:45:02.588690 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.588661    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6518f1f5_1243_49f7_8bf2_b174142e754a.slice/crio-28a6c166ce5a3049e1ea50f9a33acd4f677ab2a401b8616be4a6939a317ef8fc WatchSource:0}: Error finding container 28a6c166ce5a3049e1ea50f9a33acd4f677ab2a401b8616be4a6939a317ef8fc: Status 404 returned error can't find the container with id 28a6c166ce5a3049e1ea50f9a33acd4f677ab2a401b8616be4a6939a317ef8fc
Nov 02 14:45:02.591899 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.591870    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb407ad8b_5856_4f5f_baf7_baebc7fe9233.slice/crio-6a7dd31e0ad27d1de7f27af435f124a46464cc286cc7b71b1e637ad4bd9371cb WatchSource:0}: Error finding container 6a7dd31e0ad27d1de7f27af435f124a46464cc286cc7b71b1e637ad4bd9371cb: Status 404 returned error can't find the container with id 6a7dd31e0ad27d1de7f27af435f124a46464cc286cc7b71b1e637ad4bd9371cb
Nov 02 14:45:02.611927 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.611896    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3eed1b_a630_4ee7_b3fc_de9d967c544d.slice/crio-6c5fe916a18697f285741f0cbcbea1a0acb34baff6639772009df91c10e0316d WatchSource:0}: Error finding container 6c5fe916a18697f285741f0cbcbea1a0acb34baff6639772009df91c10e0316d: Status 404 returned error can't find the container with id 6c5fe916a18697f285741f0cbcbea1a0acb34baff6639772009df91c10e0316d
Nov 02 14:45:02.618791 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.618754    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcb0f45_1ec6_43ac_a9af_571593753a63.slice/crio-1a42536aa134dd643634630dbefdc8982e79a4d830b1c2020ffb839cff758c10 WatchSource:0}: Error finding container 1a42536aa134dd643634630dbefdc8982e79a4d830b1c2020ffb839cff758c10: Status 404 returned error can't find the container with id 1a42536aa134dd643634630dbefdc8982e79a4d830b1c2020ffb839cff758c10
Nov 02 14:45:02.620653 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.620620    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdbb368_93f7_4f30_89ef_881ec3528d6c.slice/crio-c650fa7ca021bc12907bdcaa54164771d98f1a2659585afe57eb85d3907c2171 WatchSource:0}: Error finding container c650fa7ca021bc12907bdcaa54164771d98f1a2659585afe57eb85d3907c2171: Status 404 returned error can't find the container with id c650fa7ca021bc12907bdcaa54164771d98f1a2659585afe57eb85d3907c2171
Nov 02 14:45:02.621116 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:02.621089    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63a0f7e_2b21_46b1_80d2_61d439daf047.slice/crio-918915facc6d6597cf4c19092d39650a8febbfe9f2b434b4b6d13bae66cb0f5e WatchSource:0}: Error finding container 918915facc6d6597cf4c19092d39650a8febbfe9f2b434b4b6d13bae66cb0f5e: Status 404 returned error can't find the container with id 918915facc6d6597cf4c19092d39650a8febbfe9f2b434b4b6d13bae66cb0f5e
Nov 02 14:45:03.100859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.100724    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" event={"ID":"a29378f6-902d-4efb-977a-dd1f13dd5248","Type":"ContainerStarted","Data":"5f6a4e95b24e551eeb840b70c8ce46f0ac57a6f55b2ddcbc0261643894e5f952"}
Nov 02 14:45:03.106155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.105558    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" event={"ID":"c63a0f7e-2b21-46b1-80d2-61d439daf047","Type":"ContainerStarted","Data":"918915facc6d6597cf4c19092d39650a8febbfe9f2b434b4b6d13bae66cb0f5e"}
Nov 02 14:45:03.107699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.107624    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn" event={"ID":"df67ed0b-ff8d-4774-a628-3acbf17eb2db","Type":"ContainerStarted","Data":"9ec144dcc01d9da9ea0835ffe4dd58273fdf5af22a1301332cd35d9ac110d898"}
Nov 02 14:45:03.109448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.109384    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" event={"ID":"6518f1f5-1243-49f7-8bf2-b174142e754a","Type":"ContainerStarted","Data":"28a6c166ce5a3049e1ea50f9a33acd4f677ab2a401b8616be4a6939a317ef8fc"}
Nov 02 14:45:03.113435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.112008    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" event={"ID":"ac656679-7557-495a-a88e-8b166be2da0e","Type":"ContainerStarted","Data":"53083afde1aa670b4d2ae89d7695e61661f73d101a0103a48bfb0dfd48af17ee"}
Nov 02 14:45:03.116580 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.116556    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" event={"ID":"3238cdd3-2e70-456b-afe8-6cdbf2b20b89","Type":"ContainerStarted","Data":"c152d54b2d26326414e5c3fdc0184af9efd6de905a7805fd18f804382776c68a"}
Nov 02 14:45:03.120039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.119988    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" event={"ID":"b407ad8b-5856-4f5f-baf7-baebc7fe9233","Type":"ContainerStarted","Data":"6a7dd31e0ad27d1de7f27af435f124a46464cc286cc7b71b1e637ad4bd9371cb"}
Nov 02 14:45:03.125991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.125896    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh" event={"ID":"82af8d05-2c41-441f-b905-a1b186bfcd11","Type":"ContainerStarted","Data":"d6aa318f69fef2ecef66480d54c785544c9915ec6e4114318f7336c8fd13d639"}
Nov 02 14:45:03.127961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.127938    5034 generic.go:356] "Generic (PLEG): container finished" podID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerID="736877894fd8dee085962468f779820025b855a72325fe6b143072c02bcd802a" exitCode=60
Nov 02 14:45:03.128067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.128048    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" event={"ID":"cd77d21c-5627-41d7-ab81-ec7a4651f524","Type":"ContainerDied","Data":"736877894fd8dee085962468f779820025b855a72325fe6b143072c02bcd802a"}
Nov 02 14:45:03.128303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.128285    5034 scope.go:122] "RemoveContainer" containerID="736877894fd8dee085962468f779820025b855a72325fe6b143072c02bcd802a"
Nov 02 14:45:03.131970 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.131431    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" event={"ID":"6d83cf56-9c22-47e5-aef4-087b779a2f65","Type":"ContainerStarted","Data":"42995bbf4b8e7cdc9dab8d190d7a430ab92c30f3ca32c0c9658cbe4164d06796"}
Nov 02 14:45:03.133331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.133309    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" event={"ID":"7b296f5f-1030-4705-998d-4459d0706cb0","Type":"ContainerStarted","Data":"4ea796295d39641e8cf7cc755635e47e6199fda3ee27924ef809bc1a66fb92a7"}
Nov 02 14:45:03.134772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.134745    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" event={"ID":"5b3eed1b-a630-4ee7-b3fc-de9d967c544d","Type":"ContainerStarted","Data":"6c5fe916a18697f285741f0cbcbea1a0acb34baff6639772009df91c10e0316d"}
Nov 02 14:45:03.136165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.136145    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" event={"ID":"5527c983-4d3e-4d15-8118-3f4f656b8051","Type":"ContainerStarted","Data":"6d5fe9442bdd8a972893ac7b9d21a7006d95f2d3baec5a9024665bf27aaf87bf"}
Nov 02 14:45:03.137600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.137574    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" event={"ID":"98bfda5e-d485-4a81-835b-4c369b3666e4","Type":"ContainerStarted","Data":"ca84e7847d9c5480dc8541dbaf2b9313e081c2dd049c6a980f211ca94bb133d7"}
Nov 02 14:45:03.141582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.141302    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" event={"ID":"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7","Type":"ContainerStarted","Data":"15c845092151a593d472ff5b587c91382f869fd3108aa97b89c0ccff3af488df"}
Nov 02 14:45:03.145671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.145644    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" event={"ID":"5bdbb368-93f7-4f30-89ef-881ec3528d6c","Type":"ContainerStarted","Data":"c650fa7ca021bc12907bdcaa54164771d98f1a2659585afe57eb85d3907c2171"}
Nov 02 14:45:03.147535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.147508    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8" event={"ID":"54f4abb8-f88b-42df-83c4-0dbffb278927","Type":"ContainerStarted","Data":"96066f78af50b3a20e8b08d18259f3379aa7ac91f5e6de0874d30eb3e837cc90"}
Nov 02 14:45:03.149112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.149082    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" event={"ID":"708b38e0-5f4a-4669-84e7-f74b02a3425e","Type":"ContainerStarted","Data":"5aec73989cce7f81a9e9f461e91a68e77e890ea7ce71e74c98e424eac38ec7db"}
Nov 02 14:45:03.150635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.150608    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" event={"ID":"0e9989de-be08-4242-944b-9db03854c4ed","Type":"ContainerStarted","Data":"9996c239df406912ae540b85c3369a6326e34e364e46f8893638d05bc9c5a0d2"}
Nov 02 14:45:03.152244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.152221    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" event={"ID":"86a9e4f4-79f9-4e98-be5e-f871f11b183d","Type":"ContainerStarted","Data":"7877fb7719b32c468ec4ba13ecb523e52ce3375551d660de8e20d250da822607"}
Nov 02 14:45:03.154742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.154717    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" event={"ID":"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f","Type":"ContainerStarted","Data":"1ddb4b0a9a40024ce28d61fabfbed81e280bca73f1933d4df711200fd7169c17"}
Nov 02 14:45:03.159294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.159271    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" event={"ID":"4dcb0f45-1ec6-43ac-a9af-571593753a63","Type":"ContainerStarted","Data":"1a42536aa134dd643634630dbefdc8982e79a4d830b1c2020ffb839cff758c10"}
Nov 02 14:45:03.161295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.161270    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26" event={"ID":"de34363d-7a31-4d79-8471-9b4e69d181d5","Type":"ContainerStarted","Data":"68153472f0398cf0b13df1ba7b0700cb905e2c2c28555e76ce9e5f71ab8fd917"}
Nov 02 14:45:03.162365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.162330    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" podStartSLOduration=3.162318143 podStartE2EDuration="3.162318143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.155647757 +0000 UTC m=+3693.906952197" watchObservedRunningTime="2026-11-02 14:45:03.162318143 +0000 UTC m=+3693.913622581"
Nov 02 14:45:03.232867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.232831    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz" podUID="de00d3b1-f2d5-46c7-a29c-fb1576200196" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:45:03.252881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.252814    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh" podStartSLOduration=3.252796487 podStartE2EDuration="3.252796487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.248197165 +0000 UTC m=+3693.999501603" watchObservedRunningTime="2026-11-02 14:45:03.252796487 +0000 UTC m=+3694.004100923"
Nov 02 14:45:03.294443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:03.294380    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" podStartSLOduration=3.294360195 podStartE2EDuration="3.294360195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.291666868 +0000 UTC m=+3694.042971308" watchObservedRunningTime="2026-11-02 14:45:03.294360195 +0000 UTC m=+3694.045664634"
Nov 02 14:45:04.226450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.226412    5034 generic.go:356] "Generic (PLEG): container finished" podID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerID="68153472f0398cf0b13df1ba7b0700cb905e2c2c28555e76ce9e5f71ab8fd917" exitCode=60
Nov 02 14:45:04.226592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.226556    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26" event={"ID":"de34363d-7a31-4d79-8471-9b4e69d181d5","Type":"ContainerDied","Data":"68153472f0398cf0b13df1ba7b0700cb905e2c2c28555e76ce9e5f71ab8fd917"}
Nov 02 14:45:04.234513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.231071    5034 scope.go:122] "RemoveContainer" containerID="68153472f0398cf0b13df1ba7b0700cb905e2c2c28555e76ce9e5f71ab8fd917"
Nov 02 14:45:04.235626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.235591    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" event={"ID":"a29378f6-902d-4efb-977a-dd1f13dd5248","Type":"ContainerStarted","Data":"c536989f24ace68472c4a21d718acf1df5b4788059677d9ba8db7764a0731596"}
Nov 02 14:45:04.241328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.241035    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" event={"ID":"c63a0f7e-2b21-46b1-80d2-61d439daf047","Type":"ContainerStarted","Data":"7ee09109e5e0e806c7bcb099689906e67138c08cdb67518015df870561104e36"}
Nov 02 14:45:04.255400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.255181    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn" event={"ID":"df67ed0b-ff8d-4774-a628-3acbf17eb2db","Type":"ContainerStarted","Data":"31da803eeda826ea843276e3d883183d7a2f666bf01bde61594e1667f892d5bd"}
Nov 02 14:45:04.262369 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.262330    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" event={"ID":"6518f1f5-1243-49f7-8bf2-b174142e754a","Type":"ContainerStarted","Data":"98c3142d6a6a9c34e0a009e7923b24d71318ddccb9b2b42c9a906f709fa09013"}
Nov 02 14:45:04.264422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.264393    5034 generic.go:356] "Generic (PLEG): container finished" podID="ac656679-7557-495a-a88e-8b166be2da0e" containerID="53083afde1aa670b4d2ae89d7695e61661f73d101a0103a48bfb0dfd48af17ee" exitCode=60
Nov 02 14:45:04.264575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.264474    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" event={"ID":"ac656679-7557-495a-a88e-8b166be2da0e","Type":"ContainerDied","Data":"53083afde1aa670b4d2ae89d7695e61661f73d101a0103a48bfb0dfd48af17ee"}
Nov 02 14:45:04.264867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.264845    5034 scope.go:122] "RemoveContainer" containerID="53083afde1aa670b4d2ae89d7695e61661f73d101a0103a48bfb0dfd48af17ee"
Nov 02 14:45:04.266455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.266427    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" event={"ID":"b407ad8b-5856-4f5f-baf7-baebc7fe9233","Type":"ContainerStarted","Data":"9c50143eea0404138b96059ed0586caf623c0ca500214f29f66d64092460d09e"}
Nov 02 14:45:04.273942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.269143    5034 generic.go:356] "Generic (PLEG): container finished" podID="82af8d05-2c41-441f-b905-a1b186bfcd11" containerID="d6aa318f69fef2ecef66480d54c785544c9915ec6e4114318f7336c8fd13d639" exitCode=60
Nov 02 14:45:04.273942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.269547    5034 scope.go:122] "RemoveContainer" containerID="d6aa318f69fef2ecef66480d54c785544c9915ec6e4114318f7336c8fd13d639"
Nov 02 14:45:04.273942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.273844    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh" event={"ID":"82af8d05-2c41-441f-b905-a1b186bfcd11","Type":"ContainerDied","Data":"d6aa318f69fef2ecef66480d54c785544c9915ec6e4114318f7336c8fd13d639"}
Nov 02 14:45:04.304350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.304309    5034 generic.go:356] "Generic (PLEG): container finished" podID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerID="42995bbf4b8e7cdc9dab8d190d7a430ab92c30f3ca32c0c9658cbe4164d06796" exitCode=60
Nov 02 14:45:04.304476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.304454    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" event={"ID":"6d83cf56-9c22-47e5-aef4-087b779a2f65","Type":"ContainerDied","Data":"42995bbf4b8e7cdc9dab8d190d7a430ab92c30f3ca32c0c9658cbe4164d06796"}
Nov 02 14:45:04.308943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.308890    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" event={"ID":"7b296f5f-1030-4705-998d-4459d0706cb0","Type":"ContainerStarted","Data":"1a6cc7f54e63de5e5c3b7258fd2149bdad5f2934c3db5f355cffce75ccaba265"}
Nov 02 14:45:04.313702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.313460    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" event={"ID":"5b3eed1b-a630-4ee7-b3fc-de9d967c544d","Type":"ContainerStarted","Data":"b5346e55e03eb18d1e6cec716330d1ebe8a917fe36234a1ae02eec9b92a81385"}
Nov 02 14:45:04.319857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.318906    5034 scope.go:122] "RemoveContainer" containerID="42995bbf4b8e7cdc9dab8d190d7a430ab92c30f3ca32c0c9658cbe4164d06796"
Nov 02 14:45:04.345866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.345793    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" event={"ID":"5527c983-4d3e-4d15-8118-3f4f656b8051","Type":"ContainerStarted","Data":"be388f2a42dcb3b80409960795c4254da0e70d278a08d0c699fa9784836c8876"}
Nov 02 14:45:04.351500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.351468    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" event={"ID":"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7","Type":"ContainerStarted","Data":"0911e3d927455119a7fb44bc610ab9139b1a7a3e37bd89cd595cbd41c334790a"}
Nov 02 14:45:04.354364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.354334    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" event={"ID":"5bdbb368-93f7-4f30-89ef-881ec3528d6c","Type":"ContainerStarted","Data":"09be0693741746540c67f52a489fecee696b113ccb706c6b06cf4369614a87e1"}
Nov 02 14:45:04.387099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.387061    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" event={"ID":"0e9989de-be08-4242-944b-9db03854c4ed","Type":"ContainerStarted","Data":"315efed037f7e405ee61d7eaf1985597062f47029c9e39db29f827e29a583abe"}
Nov 02 14:45:04.387512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.387490    5034 scope.go:122] "RemoveContainer" containerID="315efed037f7e405ee61d7eaf1985597062f47029c9e39db29f827e29a583abe"
Nov 02 14:45:04.392506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.392413    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" event={"ID":"86a9e4f4-79f9-4e98-be5e-f871f11b183d","Type":"ContainerStarted","Data":"6744b80269c9c0d980a63f90a0d66bbaa90cd188ed80ff7b57f9618845aa5e02"}
Nov 02 14:45:04.399024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.398492    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" event={"ID":"4dcb0f45-1ec6-43ac-a9af-571593753a63","Type":"ContainerStarted","Data":"3f1a54fb5792b73da78af4723a8c9aa135b8cb3dc05d038565cfb9ca1b9ea75d"}
Nov 02 14:45:04.420975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.420918    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" podStartSLOduration=4.420898057 podStartE2EDuration="4.420898057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.414068587 +0000 UTC m=+3695.165373025" watchObservedRunningTime="2026-11-02 14:45:04.420898057 +0000 UTC m=+3695.172202495"
Nov 02 14:45:04.459486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.459377    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn" podStartSLOduration=4.459360658 podStartE2EDuration="4.459360658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.456326182 +0000 UTC m=+3695.207630621" watchObservedRunningTime="2026-11-02 14:45:04.459360658 +0000 UTC m=+3695.210665096"
Nov 02 14:45:04.552197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.552143    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" podStartSLOduration=4.552128871 podStartE2EDuration="4.552128871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.548415771 +0000 UTC m=+3695.299720211" watchObservedRunningTime="2026-11-02 14:45:04.552128871 +0000 UTC m=+3695.303433308"
Nov 02 14:45:04.596500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.596443    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" podStartSLOduration=4.59642774 podStartE2EDuration="4.59642774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.592844173 +0000 UTC m=+3695.344148612" watchObservedRunningTime="2026-11-02 14:45:04.59642774 +0000 UTC m=+3695.347732218"
Nov 02 14:45:04.638270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.638214    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" podStartSLOduration=4.638198011 podStartE2EDuration="4.638198011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.636510466 +0000 UTC m=+3695.387814906" watchObservedRunningTime="2026-11-02 14:45:04.638198011 +0000 UTC m=+3695.389502448"
Nov 02 14:45:04.730389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.730336    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" podStartSLOduration=4.730321395 podStartE2EDuration="4.730321395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.727941236 +0000 UTC m=+3695.479245675" watchObservedRunningTime="2026-11-02 14:45:04.730321395 +0000 UTC m=+3695.481625833"
Nov 02 14:45:04.732403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.732266    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" podStartSLOduration=4.73225038 podStartE2EDuration="4.73225038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.68574235 +0000 UTC m=+3695.437046790" watchObservedRunningTime="2026-11-02 14:45:04.73225038 +0000 UTC m=+3695.483554821"
Nov 02 14:45:04.773974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.773609    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8" podStartSLOduration=4.773590154 podStartE2EDuration="4.773590154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.773201894 +0000 UTC m=+3695.524506335" watchObservedRunningTime="2026-11-02 14:45:04.773590154 +0000 UTC m=+3695.524894594"
Nov 02 14:45:04.906820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.906732    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" podStartSLOduration=4.906716221 podStartE2EDuration="4.906716221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.903524804 +0000 UTC m=+3695.654829255" watchObservedRunningTime="2026-11-02 14:45:04.906716221 +0000 UTC m=+3695.658020660"
Nov 02 14:45:04.952255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.952185    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" podStartSLOduration=4.952166892 podStartE2EDuration="4.952166892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.946882936 +0000 UTC m=+3695.698187377" watchObservedRunningTime="2026-11-02 14:45:04.952166892 +0000 UTC m=+3695.703471331"
Nov 02 14:45:04.990995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:04.990929    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" podStartSLOduration=4.990910513 podStartE2EDuration="4.990910513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:04.989551134 +0000 UTC m=+3695.740855573" watchObservedRunningTime="2026-11-02 14:45:04.990910513 +0000 UTC m=+3695.742214954"
Nov 02 14:45:05.041088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.041030    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" podStartSLOduration=5.041011541 podStartE2EDuration="5.041011541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:05.034303972 +0000 UTC m=+3695.785608411" watchObservedRunningTime="2026-11-02 14:45:05.041011541 +0000 UTC m=+3695.792315973"
Nov 02 14:45:05.081112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.081038    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" podStartSLOduration=5.081016997 podStartE2EDuration="5.081016997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:05.078314293 +0000 UTC m=+3695.829618733" watchObservedRunningTime="2026-11-02 14:45:05.081016997 +0000 UTC m=+3695.832321435"
Nov 02 14:45:05.413797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.413743    5034 generic.go:356] "Generic (PLEG): container finished" podID="7b296f5f-1030-4705-998d-4459d0706cb0" containerID="1a6cc7f54e63de5e5c3b7258fd2149bdad5f2934c3db5f355cffce75ccaba265" exitCode=60
Nov 02 14:45:05.414206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.413892    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" event={"ID":"7b296f5f-1030-4705-998d-4459d0706cb0","Type":"ContainerDied","Data":"1a6cc7f54e63de5e5c3b7258fd2149bdad5f2934c3db5f355cffce75ccaba265"}
Nov 02 14:45:05.414470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.414444    5034 scope.go:122] "RemoveContainer" containerID="1a6cc7f54e63de5e5c3b7258fd2149bdad5f2934c3db5f355cffce75ccaba265"
Nov 02 14:45:05.421211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.420352    5034 generic.go:356] "Generic (PLEG): container finished" podID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerID="b5346e55e03eb18d1e6cec716330d1ebe8a917fe36234a1ae02eec9b92a81385" exitCode=60
Nov 02 14:45:05.421211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.420388    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" event={"ID":"5b3eed1b-a630-4ee7-b3fc-de9d967c544d","Type":"ContainerDied","Data":"b5346e55e03eb18d1e6cec716330d1ebe8a917fe36234a1ae02eec9b92a81385"}
Nov 02 14:45:05.421211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.420773    5034 scope.go:122] "RemoveContainer" containerID="b5346e55e03eb18d1e6cec716330d1ebe8a917fe36234a1ae02eec9b92a81385"
Nov 02 14:45:05.423561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.423536    5034 generic.go:356] "Generic (PLEG): container finished" podID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerID="be388f2a42dcb3b80409960795c4254da0e70d278a08d0c699fa9784836c8876" exitCode=60
Nov 02 14:45:05.423702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.423565    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" event={"ID":"5527c983-4d3e-4d15-8118-3f4f656b8051","Type":"ContainerDied","Data":"be388f2a42dcb3b80409960795c4254da0e70d278a08d0c699fa9784836c8876"}
Nov 02 14:45:05.424012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.423989    5034 scope.go:122] "RemoveContainer" containerID="be388f2a42dcb3b80409960795c4254da0e70d278a08d0c699fa9784836c8876"
Nov 02 14:45:05.433346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.433317    5034 generic.go:356] "Generic (PLEG): container finished" podID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerID="8bff722ab42aef86949e99aac06488557829eceb1978bc7eefc13675f9e1d643" exitCode=60
Nov 02 14:45:05.433467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.433441    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" event={"ID":"98bfda5e-d485-4a81-835b-4c369b3666e4","Type":"ContainerDied","Data":"8bff722ab42aef86949e99aac06488557829eceb1978bc7eefc13675f9e1d643"}
Nov 02 14:45:05.433853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.433827    5034 scope.go:122] "RemoveContainer" containerID="8bff722ab42aef86949e99aac06488557829eceb1978bc7eefc13675f9e1d643"
Nov 02 14:45:05.449634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.449598    5034 generic.go:356] "Generic (PLEG): container finished" podID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerID="0911e3d927455119a7fb44bc610ab9139b1a7a3e37bd89cd595cbd41c334790a" exitCode=60
Nov 02 14:45:05.449981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.449680    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" event={"ID":"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7","Type":"ContainerDied","Data":"0911e3d927455119a7fb44bc610ab9139b1a7a3e37bd89cd595cbd41c334790a"}
Nov 02 14:45:05.450170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.450148    5034 scope.go:122] "RemoveContainer" containerID="0911e3d927455119a7fb44bc610ab9139b1a7a3e37bd89cd595cbd41c334790a"
Nov 02 14:45:05.457732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.457697    5034 generic.go:356] "Generic (PLEG): container finished" podID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerID="09be0693741746540c67f52a489fecee696b113ccb706c6b06cf4369614a87e1" exitCode=60
Nov 02 14:45:05.457895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.457858    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" event={"ID":"5bdbb368-93f7-4f30-89ef-881ec3528d6c","Type":"ContainerDied","Data":"09be0693741746540c67f52a489fecee696b113ccb706c6b06cf4369614a87e1"}
Nov 02 14:45:05.458436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.458409    5034 scope.go:122] "RemoveContainer" containerID="09be0693741746540c67f52a489fecee696b113ccb706c6b06cf4369614a87e1"
Nov 02 14:45:05.461508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.461478    5034 generic.go:356] "Generic (PLEG): container finished" podID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerID="9dd35028fd14cd94f3fa09692496650949e5d067d7333dcbcc95f57f56d88341" exitCode=60
Nov 02 14:45:05.461620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.461590    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8" event={"ID":"54f4abb8-f88b-42df-83c4-0dbffb278927","Type":"ContainerDied","Data":"9dd35028fd14cd94f3fa09692496650949e5d067d7333dcbcc95f57f56d88341"}
Nov 02 14:45:05.462052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.462031    5034 scope.go:122] "RemoveContainer" containerID="9dd35028fd14cd94f3fa09692496650949e5d067d7333dcbcc95f57f56d88341"
Nov 02 14:45:05.464693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.464665    5034 generic.go:356] "Generic (PLEG): container finished" podID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerID="9abd75a0a3064dfd7b6c34ffe3672104b64c83146bf82a695730db912c3c048c" exitCode=60
Nov 02 14:45:05.464802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.464752    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" event={"ID":"708b38e0-5f4a-4669-84e7-f74b02a3425e","Type":"ContainerDied","Data":"9abd75a0a3064dfd7b6c34ffe3672104b64c83146bf82a695730db912c3c048c"}
Nov 02 14:45:05.465073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.465059    5034 scope.go:122] "RemoveContainer" containerID="9abd75a0a3064dfd7b6c34ffe3672104b64c83146bf82a695730db912c3c048c"
Nov 02 14:45:05.474438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.474409    5034 generic.go:356] "Generic (PLEG): container finished" podID="0e9989de-be08-4242-944b-9db03854c4ed" containerID="315efed037f7e405ee61d7eaf1985597062f47029c9e39db29f827e29a583abe" exitCode=60
Nov 02 14:45:05.474541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.474524    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" event={"ID":"0e9989de-be08-4242-944b-9db03854c4ed","Type":"ContainerDied","Data":"315efed037f7e405ee61d7eaf1985597062f47029c9e39db29f827e29a583abe"}
Nov 02 14:45:05.480982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.480593    5034 generic.go:356] "Generic (PLEG): container finished" podID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerID="6744b80269c9c0d980a63f90a0d66bbaa90cd188ed80ff7b57f9618845aa5e02" exitCode=60
Nov 02 14:45:05.480982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.480703    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" event={"ID":"86a9e4f4-79f9-4e98-be5e-f871f11b183d","Type":"ContainerDied","Data":"6744b80269c9c0d980a63f90a0d66bbaa90cd188ed80ff7b57f9618845aa5e02"}
Nov 02 14:45:05.483499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.483467    5034 generic.go:356] "Generic (PLEG): container finished" podID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerID="53f6e01c2fa1bc68a11db9c6f754a6621bdfb4fd76f98eb72ebe0bcb4faa5735" exitCode=60
Nov 02 14:45:05.483645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.483594    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" event={"ID":"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f","Type":"ContainerDied","Data":"53f6e01c2fa1bc68a11db9c6f754a6621bdfb4fd76f98eb72ebe0bcb4faa5735"}
Nov 02 14:45:05.490881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.490839    5034 scope.go:122] "RemoveContainer" containerID="53f6e01c2fa1bc68a11db9c6f754a6621bdfb4fd76f98eb72ebe0bcb4faa5735"
Nov 02 14:45:05.492884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.492765    5034 generic.go:356] "Generic (PLEG): container finished" podID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerID="3f1a54fb5792b73da78af4723a8c9aa135b8cb3dc05d038565cfb9ca1b9ea75d" exitCode=60
Nov 02 14:45:05.493032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.492980    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" event={"ID":"4dcb0f45-1ec6-43ac-a9af-571593753a63","Type":"ContainerDied","Data":"3f1a54fb5792b73da78af4723a8c9aa135b8cb3dc05d038565cfb9ca1b9ea75d"}
Nov 02 14:45:05.493519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.493300    5034 scope.go:122] "RemoveContainer" containerID="3f1a54fb5792b73da78af4723a8c9aa135b8cb3dc05d038565cfb9ca1b9ea75d"
Nov 02 14:45:05.500157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.499958    5034 scope.go:122] "RemoveContainer" containerID="6744b80269c9c0d980a63f90a0d66bbaa90cd188ed80ff7b57f9618845aa5e02"
Nov 02 14:45:05.517881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.517803    5034 generic.go:356] "Generic (PLEG): container finished" podID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerID="c536989f24ace68472c4a21d718acf1df5b4788059677d9ba8db7764a0731596" exitCode=60
Nov 02 14:45:05.521686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.521648    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" event={"ID":"a29378f6-902d-4efb-977a-dd1f13dd5248","Type":"ContainerDied","Data":"c536989f24ace68472c4a21d718acf1df5b4788059677d9ba8db7764a0731596"}
Nov 02 14:45:05.522622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.522596    5034 scope.go:122] "RemoveContainer" containerID="c536989f24ace68472c4a21d718acf1df5b4788059677d9ba8db7764a0731596"
Nov 02 14:45:05.537626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.537592    5034 generic.go:356] "Generic (PLEG): container finished" podID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerID="7ee09109e5e0e806c7bcb099689906e67138c08cdb67518015df870561104e36" exitCode=60
Nov 02 14:45:05.537957 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.537932    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" event={"ID":"c63a0f7e-2b21-46b1-80d2-61d439daf047","Type":"ContainerDied","Data":"7ee09109e5e0e806c7bcb099689906e67138c08cdb67518015df870561104e36"}
Nov 02 14:45:05.538659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.538636    5034 scope.go:122] "RemoveContainer" containerID="7ee09109e5e0e806c7bcb099689906e67138c08cdb67518015df870561104e36"
Nov 02 14:45:05.542862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.542553    5034 generic.go:356] "Generic (PLEG): container finished" podID="df67ed0b-ff8d-4774-a628-3acbf17eb2db" containerID="31da803eeda826ea843276e3d883183d7a2f666bf01bde61594e1667f892d5bd" exitCode=60
Nov 02 14:45:05.542862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.542648    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn" event={"ID":"df67ed0b-ff8d-4774-a628-3acbf17eb2db","Type":"ContainerDied","Data":"31da803eeda826ea843276e3d883183d7a2f666bf01bde61594e1667f892d5bd"}
Nov 02 14:45:05.545184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.543991    5034 scope.go:122] "RemoveContainer" containerID="31da803eeda826ea843276e3d883183d7a2f666bf01bde61594e1667f892d5bd"
Nov 02 14:45:05.546497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.545867    5034 generic.go:356] "Generic (PLEG): container finished" podID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerID="98c3142d6a6a9c34e0a009e7923b24d71318ddccb9b2b42c9a906f709fa09013" exitCode=60
Nov 02 14:45:05.546497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.546029    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" event={"ID":"6518f1f5-1243-49f7-8bf2-b174142e754a","Type":"ContainerDied","Data":"98c3142d6a6a9c34e0a009e7923b24d71318ddccb9b2b42c9a906f709fa09013"}
Nov 02 14:45:05.546668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.546502    5034 scope.go:122] "RemoveContainer" containerID="98c3142d6a6a9c34e0a009e7923b24d71318ddccb9b2b42c9a906f709fa09013"
Nov 02 14:45:05.583175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.583139    5034 generic.go:356] "Generic (PLEG): container finished" podID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerID="43fa0641e208992d15a139ad17495567b476cb91d08c701703077f7a1fcf4a1c" exitCode=60
Nov 02 14:45:05.583309 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.583244    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" event={"ID":"3238cdd3-2e70-456b-afe8-6cdbf2b20b89","Type":"ContainerDied","Data":"43fa0641e208992d15a139ad17495567b476cb91d08c701703077f7a1fcf4a1c"}
Nov 02 14:45:05.583761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.583735    5034 scope.go:122] "RemoveContainer" containerID="43fa0641e208992d15a139ad17495567b476cb91d08c701703077f7a1fcf4a1c"
Nov 02 14:45:05.593823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.593223    5034 generic.go:356] "Generic (PLEG): container finished" podID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerID="9c50143eea0404138b96059ed0586caf623c0ca500214f29f66d64092460d09e" exitCode=60
Nov 02 14:45:05.593823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.593352    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" event={"ID":"b407ad8b-5856-4f5f-baf7-baebc7fe9233","Type":"ContainerDied","Data":"9c50143eea0404138b96059ed0586caf623c0ca500214f29f66d64092460d09e"}
Nov 02 14:45:05.594005 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.593933    5034 scope.go:122] "RemoveContainer" containerID="9c50143eea0404138b96059ed0586caf623c0ca500214f29f66d64092460d09e"
Nov 02 14:45:05.597436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.597411    5034 generic.go:356] "Generic (PLEG): container finished" podID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerID="45cfa36a84f7ee6ff73352166935312eeee9f7542dbac1f1b709011d9975b20f" exitCode=60
Nov 02 14:45:05.597623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.597459    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" event={"ID":"cd77d21c-5627-41d7-ab81-ec7a4651f524","Type":"ContainerDied","Data":"45cfa36a84f7ee6ff73352166935312eeee9f7542dbac1f1b709011d9975b20f"}
Nov 02 14:45:05.597623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.597515    5034 scope.go:122] "RemoveContainer" containerID="736877894fd8dee085962468f779820025b855a72325fe6b143072c02bcd802a"
Nov 02 14:45:05.598112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:05.598094    5034 scope.go:122] "RemoveContainer" containerID="45cfa36a84f7ee6ff73352166935312eeee9f7542dbac1f1b709011d9975b20f"
Nov 02 14:45:05.598493 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:05.598471    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp_odh-ai-gateway-infra(cd77d21c-5627-41d7-ab81-ec7a4651f524)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524"
Nov 02 14:45:07.338802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:07.298869    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:45:07.454159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:07.406920    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:45:07.573679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:07.573635    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:45:07.574246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:07.574218    5034 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:45:07.609823 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:07.609755    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" event={"ID":"ac656679-7557-495a-a88e-8b166be2da0e","Type":"ContainerStarted","Data":"6657a641f4f7a5482ee8906da993445b6c25769fec607b7fa2a620427031138f"}
Nov 02 14:45:07.610150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:07.610109    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh" event={"ID":"82af8d05-2c41-441f-b905-a1b186bfcd11","Type":"ContainerStarted","Data":"c78547df16d4f9c4c2eb79e48e2a9e07c62f0860c13b02ffa1f31a934ef286ba"}
Nov 02 14:45:07.626129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:07.626098    5034 scope.go:122] "RemoveContainer" containerID="45cfa36a84f7ee6ff73352166935312eeee9f7542dbac1f1b709011d9975b20f"
Nov 02 14:45:07.626657 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:07.626627    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp_odh-ai-gateway-infra(cd77d21c-5627-41d7-ab81-ec7a4651f524)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524"
Nov 02 14:45:07.640923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:07.640762    5034 scope.go:122] "RemoveContainer" containerID="0af883b6767615d6bab52a43bbb6b84848945b3dc4bbac77303ea777eb070f9c"
Nov 02 14:45:07.641351 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:07.641310    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr_odh-ai-gateway-infra(7b296f5f-1030-4705-998d-4459d0706cb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" podUID="7b296f5f-1030-4705-998d-4459d0706cb0"
Nov 02 14:45:07.814305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:07.812814    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26" podStartSLOduration=7.812772132 podStartE2EDuration="7.812772132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:03.340043732 +0000 UTC m=+3694.091348171" watchObservedRunningTime="2026-11-02 14:45:07.812772132 +0000 UTC m=+3698.564076548"
Nov 02 14:45:08.764161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.764126    5034 generic.go:356] "Generic (PLEG): container finished" podID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerID="3da29a98819607edd645993da30b78da4d5839d1f56af227c9e88d35e38c877c" exitCode=60
Nov 02 14:45:08.764637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.764250    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26" event={"ID":"de34363d-7a31-4d79-8471-9b4e69d181d5","Type":"ContainerDied","Data":"3da29a98819607edd645993da30b78da4d5839d1f56af227c9e88d35e38c877c"}
Nov 02 14:45:08.764637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.764350    5034 scope.go:122] "RemoveContainer" containerID="68153472f0398cf0b13df1ba7b0700cb905e2c2c28555e76ce9e5f71ab8fd917"
Nov 02 14:45:08.764962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.764943    5034 scope.go:122] "RemoveContainer" containerID="3da29a98819607edd645993da30b78da4d5839d1f56af227c9e88d35e38c877c"
Nov 02 14:45:08.765969 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.765407    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26_odh-ai-gateway-infra(de34363d-7a31-4d79-8471-9b4e69d181d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5"
Nov 02 14:45:08.767841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.767817    5034 generic.go:356] "Generic (PLEG): container finished" podID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerID="b81d119011cf51f51cc04c677154c8ccb5565f41c74ae32150f57561e060c27e" exitCode=60
Nov 02 14:45:08.767958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.767907    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" event={"ID":"a29378f6-902d-4efb-977a-dd1f13dd5248","Type":"ContainerDied","Data":"b81d119011cf51f51cc04c677154c8ccb5565f41c74ae32150f57561e060c27e"}
Nov 02 14:45:08.768174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.768158    5034 scope.go:122] "RemoveContainer" containerID="b81d119011cf51f51cc04c677154c8ccb5565f41c74ae32150f57561e060c27e"
Nov 02 14:45:08.768558 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.768538    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn_odh-ai-gateway-infra(a29378f6-902d-4efb-977a-dd1f13dd5248)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248"
Nov 02 14:45:08.772752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.772720    5034 generic.go:356] "Generic (PLEG): container finished" podID="df67ed0b-ff8d-4774-a628-3acbf17eb2db" containerID="ddb44fe7265e7fa3dc61f5dc6574973023420a8309e992eddd4732d08bc45bdc" exitCode=60
Nov 02 14:45:08.772939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.772853    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn" event={"ID":"df67ed0b-ff8d-4774-a628-3acbf17eb2db","Type":"ContainerDied","Data":"ddb44fe7265e7fa3dc61f5dc6574973023420a8309e992eddd4732d08bc45bdc"}
Nov 02 14:45:08.773260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.773238    5034 scope.go:122] "RemoveContainer" containerID="ddb44fe7265e7fa3dc61f5dc6574973023420a8309e992eddd4732d08bc45bdc"
Nov 02 14:45:08.773766 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.773737    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn_odh-ai-gateway-infra(df67ed0b-ff8d-4774-a628-3acbf17eb2db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn" podUID="df67ed0b-ff8d-4774-a628-3acbf17eb2db"
Nov 02 14:45:08.778175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.778116    5034 generic.go:356] "Generic (PLEG): container finished" podID="ac656679-7557-495a-a88e-8b166be2da0e" containerID="6657a641f4f7a5482ee8906da993445b6c25769fec607b7fa2a620427031138f" exitCode=60
Nov 02 14:45:08.778279 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.778194    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" event={"ID":"ac656679-7557-495a-a88e-8b166be2da0e","Type":"ContainerDied","Data":"6657a641f4f7a5482ee8906da993445b6c25769fec607b7fa2a620427031138f"}
Nov 02 14:45:08.778499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.778482    5034 scope.go:122] "RemoveContainer" containerID="6657a641f4f7a5482ee8906da993445b6c25769fec607b7fa2a620427031138f"
Nov 02 14:45:08.778943 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.778918    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt_odh-ai-gateway-infra(ac656679-7557-495a-a88e-8b166be2da0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" podUID="ac656679-7557-495a-a88e-8b166be2da0e"
Nov 02 14:45:08.780339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.780314    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" event={"ID":"3238cdd3-2e70-456b-afe8-6cdbf2b20b89","Type":"ContainerStarted","Data":"fa0819fba1215530776f58a8a21fbddc2e67a01804957d7c1c2650f59beffaf1"}
Nov 02 14:45:08.782372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.782351    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" event={"ID":"b407ad8b-5856-4f5f-baf7-baebc7fe9233","Type":"ContainerStarted","Data":"476ec33a22a342d6905570e705e2010e8b9d54a3a1db9203fe8f4543c1884edb"}
Nov 02 14:45:08.784327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.784304    5034 generic.go:356] "Generic (PLEG): container finished" podID="82af8d05-2c41-441f-b905-a1b186bfcd11" containerID="c78547df16d4f9c4c2eb79e48e2a9e07c62f0860c13b02ffa1f31a934ef286ba" exitCode=60
Nov 02 14:45:08.784422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.784404    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh" event={"ID":"82af8d05-2c41-441f-b905-a1b186bfcd11","Type":"ContainerDied","Data":"c78547df16d4f9c4c2eb79e48e2a9e07c62f0860c13b02ffa1f31a934ef286ba"}
Nov 02 14:45:08.784597 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.784582    5034 scope.go:122] "RemoveContainer" containerID="c78547df16d4f9c4c2eb79e48e2a9e07c62f0860c13b02ffa1f31a934ef286ba"
Nov 02 14:45:08.784929 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.784910    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh_odh-ai-gateway-infra(82af8d05-2c41-441f-b905-a1b186bfcd11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh" podUID="82af8d05-2c41-441f-b905-a1b186bfcd11"
Nov 02 14:45:08.786293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.786273    5034 generic.go:356] "Generic (PLEG): container finished" podID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerID="ad2bdef45e40871ec0f623c6390af9c9175e021658ccdf4995ca09a0f8f852f3" exitCode=60
Nov 02 14:45:08.786365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.786350    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" event={"ID":"6d83cf56-9c22-47e5-aef4-087b779a2f65","Type":"ContainerDied","Data":"ad2bdef45e40871ec0f623c6390af9c9175e021658ccdf4995ca09a0f8f852f3"}
Nov 02 14:45:08.786604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.786585    5034 scope.go:122] "RemoveContainer" containerID="ad2bdef45e40871ec0f623c6390af9c9175e021658ccdf4995ca09a0f8f852f3"
Nov 02 14:45:08.786958 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.786935    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr_odh-ai-gateway-infra(6d83cf56-9c22-47e5-aef4-087b779a2f65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65"
Nov 02 14:45:08.788350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.788333    5034 generic.go:356] "Generic (PLEG): container finished" podID="7b296f5f-1030-4705-998d-4459d0706cb0" containerID="0af883b6767615d6bab52a43bbb6b84848945b3dc4bbac77303ea777eb070f9c" exitCode=60
Nov 02 14:45:08.788436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.788402    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" event={"ID":"7b296f5f-1030-4705-998d-4459d0706cb0","Type":"ContainerDied","Data":"0af883b6767615d6bab52a43bbb6b84848945b3dc4bbac77303ea777eb070f9c"}
Nov 02 14:45:08.788731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.788714    5034 scope.go:122] "RemoveContainer" containerID="0af883b6767615d6bab52a43bbb6b84848945b3dc4bbac77303ea777eb070f9c"
Nov 02 14:45:08.789032 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.789013    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr_odh-ai-gateway-infra(7b296f5f-1030-4705-998d-4459d0706cb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" podUID="7b296f5f-1030-4705-998d-4459d0706cb0"
Nov 02 14:45:08.790365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.790347    5034 generic.go:356] "Generic (PLEG): container finished" podID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerID="609b428e1321080ba0125383f79779ce35ff073ad567c7372983ac935434201f" exitCode=60
Nov 02 14:45:08.790478 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.790405    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" event={"ID":"5b3eed1b-a630-4ee7-b3fc-de9d967c544d","Type":"ContainerDied","Data":"609b428e1321080ba0125383f79779ce35ff073ad567c7372983ac935434201f"}
Nov 02 14:45:08.790672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.790654    5034 scope.go:122] "RemoveContainer" containerID="609b428e1321080ba0125383f79779ce35ff073ad567c7372983ac935434201f"
Nov 02 14:45:08.790970 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.790948    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj_odh-ai-gateway-infra(5b3eed1b-a630-4ee7-b3fc-de9d967c544d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d"
Nov 02 14:45:08.792463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.792441    5034 generic.go:356] "Generic (PLEG): container finished" podID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerID="8d47d5cd66be15c1eb5f773696d0863e3de5cb5d0276f054ab20dc8a6dc71808" exitCode=60
Nov 02 14:45:08.792546 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.792517    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" event={"ID":"5527c983-4d3e-4d15-8118-3f4f656b8051","Type":"ContainerDied","Data":"8d47d5cd66be15c1eb5f773696d0863e3de5cb5d0276f054ab20dc8a6dc71808"}
Nov 02 14:45:08.792749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.792734    5034 scope.go:122] "RemoveContainer" containerID="8d47d5cd66be15c1eb5f773696d0863e3de5cb5d0276f054ab20dc8a6dc71808"
Nov 02 14:45:08.793047 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.793025    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb_odh-ai-gateway-infra(5527c983-4d3e-4d15-8118-3f4f656b8051)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051"
Nov 02 14:45:08.794441 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.794420    5034 generic.go:356] "Generic (PLEG): container finished" podID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerID="21b1155736f2f97d9206a0cb056998e08fe770a233a2cb4b5bafc23e33654456" exitCode=60
Nov 02 14:45:08.794527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.794481    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" event={"ID":"98bfda5e-d485-4a81-835b-4c369b3666e4","Type":"ContainerDied","Data":"21b1155736f2f97d9206a0cb056998e08fe770a233a2cb4b5bafc23e33654456"}
Nov 02 14:45:08.794730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.794714    5034 scope.go:122] "RemoveContainer" containerID="21b1155736f2f97d9206a0cb056998e08fe770a233a2cb4b5bafc23e33654456"
Nov 02 14:45:08.794982 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.794963    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq_odh-ai-gateway-infra(98bfda5e-d485-4a81-835b-4c369b3666e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4"
Nov 02 14:45:08.796397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.796374    5034 generic.go:356] "Generic (PLEG): container finished" podID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerID="3c1a727b903e55fef6636bc699c1d354af0e3903cf2a120b78015d6e049a41b6" exitCode=60
Nov 02 14:45:08.796504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.796451    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" event={"ID":"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7","Type":"ContainerDied","Data":"3c1a727b903e55fef6636bc699c1d354af0e3903cf2a120b78015d6e049a41b6"}
Nov 02 14:45:08.796713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.796697    5034 scope.go:122] "RemoveContainer" containerID="3c1a727b903e55fef6636bc699c1d354af0e3903cf2a120b78015d6e049a41b6"
Nov 02 14:45:08.796976 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.796949    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2_odh-ai-gateway-infra(c7ee5b3f-068d-40b7-aaf1-461643ff7fe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7"
Nov 02 14:45:08.798272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.798255    5034 generic.go:356] "Generic (PLEG): container finished" podID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerID="d0d0c1abafeb962fb562ca120517b253528fd967d06fd94889b79e38b8afb8f8" exitCode=60
Nov 02 14:45:08.798348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.798318    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" event={"ID":"5bdbb368-93f7-4f30-89ef-881ec3528d6c","Type":"ContainerDied","Data":"d0d0c1abafeb962fb562ca120517b253528fd967d06fd94889b79e38b8afb8f8"}
Nov 02 14:45:08.798612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.798592    5034 scope.go:122] "RemoveContainer" containerID="d0d0c1abafeb962fb562ca120517b253528fd967d06fd94889b79e38b8afb8f8"
Nov 02 14:45:08.798944 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.798916    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn_odh-ai-gateway-infra(5bdbb368-93f7-4f30-89ef-881ec3528d6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c"
Nov 02 14:45:08.801245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.801225    5034 generic.go:356] "Generic (PLEG): container finished" podID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerID="df33e2b42bd38d830f53eda889ecc2d5587e23c2adcfc95035c7dda85eb97ac9" exitCode=60
Nov 02 14:45:08.801354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.801295    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8" event={"ID":"54f4abb8-f88b-42df-83c4-0dbffb278927","Type":"ContainerDied","Data":"df33e2b42bd38d830f53eda889ecc2d5587e23c2adcfc95035c7dda85eb97ac9"}
Nov 02 14:45:08.801615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.801601    5034 scope.go:122] "RemoveContainer" containerID="df33e2b42bd38d830f53eda889ecc2d5587e23c2adcfc95035c7dda85eb97ac9"
Nov 02 14:45:08.801930 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.801898    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8_odh-ai-gateway-infra(54f4abb8-f88b-42df-83c4-0dbffb278927)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927"
Nov 02 14:45:08.803507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.803460    5034 generic.go:356] "Generic (PLEG): container finished" podID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerID="d0385e342fedf82711eda835ecf557bcc3abbe42e0ef244e5c471edde568d51f" exitCode=60
Nov 02 14:45:08.803507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.803488    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" event={"ID":"708b38e0-5f4a-4669-84e7-f74b02a3425e","Type":"ContainerDied","Data":"d0385e342fedf82711eda835ecf557bcc3abbe42e0ef244e5c471edde568d51f"}
Nov 02 14:45:08.803833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.803773    5034 scope.go:122] "RemoveContainer" containerID="d0385e342fedf82711eda835ecf557bcc3abbe42e0ef244e5c471edde568d51f"
Nov 02 14:45:08.804098 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.804073    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc_odh-ai-gateway-infra(708b38e0-5f4a-4669-84e7-f74b02a3425e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e"
Nov 02 14:45:08.806521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.806497    5034 generic.go:356] "Generic (PLEG): container finished" podID="0e9989de-be08-4242-944b-9db03854c4ed" containerID="a4b81ca7474705a5a7ae8c8ef357d04849c2ae94c5cd7beaaf656e95c17fe748" exitCode=60
Nov 02 14:45:08.806642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.806521    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" event={"ID":"0e9989de-be08-4242-944b-9db03854c4ed","Type":"ContainerDied","Data":"a4b81ca7474705a5a7ae8c8ef357d04849c2ae94c5cd7beaaf656e95c17fe748"}
Nov 02 14:45:08.806828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.806810    5034 scope.go:122] "RemoveContainer" containerID="a4b81ca7474705a5a7ae8c8ef357d04849c2ae94c5cd7beaaf656e95c17fe748"
Nov 02 14:45:08.807090 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:08.807069    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv_odh-ai-gateway-infra(0e9989de-be08-4242-944b-9db03854c4ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" podUID="0e9989de-be08-4242-944b-9db03854c4ed"
Nov 02 14:45:08.808979 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.808956    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" event={"ID":"86a9e4f4-79f9-4e98-be5e-f871f11b183d","Type":"ContainerStarted","Data":"e72a369c122d8ede1688d22861b4d15a19d55c9f516f9e64848208ac0d3f7d5f"}
Nov 02 14:45:08.811190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.811155    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" event={"ID":"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f","Type":"ContainerStarted","Data":"69d49dc46e78620954bae32579243fb930d5501080e62acf8ed1dd0428b8a19c"}
Nov 02 14:45:08.813181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:08.813161    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" event={"ID":"4dcb0f45-1ec6-43ac-a9af-571593753a63","Type":"ContainerStarted","Data":"cf7660caa42c704c5fb88b9d27cfccd0b18bc63676346559d5065d0ee77331d2"}
Nov 02 14:45:09.113494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.113428    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" podStartSLOduration=9.113409632 podStartE2EDuration="9.113409632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:09.111539175 +0000 UTC m=+3699.862843613" watchObservedRunningTime="2026-11-02 14:45:09.113409632 +0000 UTC m=+3699.864714071"
Nov 02 14:45:09.380606 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.380584    5034 scope.go:122] "RemoveContainer" containerID="c536989f24ace68472c4a21d718acf1df5b4788059677d9ba8db7764a0731596"
Nov 02 14:45:09.400170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.400142    5034 scope.go:122] "RemoveContainer" containerID="31da803eeda826ea843276e3d883183d7a2f666bf01bde61594e1667f892d5bd"
Nov 02 14:45:09.416223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.416201    5034 scope.go:122] "RemoveContainer" containerID="53083afde1aa670b4d2ae89d7695e61661f73d101a0103a48bfb0dfd48af17ee"
Nov 02 14:45:09.469465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.469440    5034 scope.go:122] "RemoveContainer" containerID="d6aa318f69fef2ecef66480d54c785544c9915ec6e4114318f7336c8fd13d639"
Nov 02 14:45:09.493820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.493773    5034 scope.go:122] "RemoveContainer" containerID="42995bbf4b8e7cdc9dab8d190d7a430ab92c30f3ca32c0c9658cbe4164d06796"
Nov 02 14:45:09.510853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.510821    5034 scope.go:122] "RemoveContainer" containerID="1a6cc7f54e63de5e5c3b7258fd2149bdad5f2934c3db5f355cffce75ccaba265"
Nov 02 14:45:09.530491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.530469    5034 scope.go:122] "RemoveContainer" containerID="b5346e55e03eb18d1e6cec716330d1ebe8a917fe36234a1ae02eec9b92a81385"
Nov 02 14:45:09.546624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.546595    5034 scope.go:122] "RemoveContainer" containerID="be388f2a42dcb3b80409960795c4254da0e70d278a08d0c699fa9784836c8876"
Nov 02 14:45:09.562821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.562794    5034 scope.go:122] "RemoveContainer" containerID="8bff722ab42aef86949e99aac06488557829eceb1978bc7eefc13675f9e1d643"
Nov 02 14:45:09.592636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.592607    5034 scope.go:122] "RemoveContainer" containerID="0911e3d927455119a7fb44bc610ab9139b1a7a3e37bd89cd595cbd41c334790a"
Nov 02 14:45:09.611757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.611727    5034 scope.go:122] "RemoveContainer" containerID="09be0693741746540c67f52a489fecee696b113ccb706c6b06cf4369614a87e1"
Nov 02 14:45:09.615458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.615405    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" podStartSLOduration=9.615393078 podStartE2EDuration="9.615393078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:09.613617861 +0000 UTC m=+3700.364922310" watchObservedRunningTime="2026-11-02 14:45:09.615393078 +0000 UTC m=+3700.366697515"
Nov 02 14:45:09.628931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.628902    5034 scope.go:122] "RemoveContainer" containerID="9dd35028fd14cd94f3fa09692496650949e5d067d7333dcbcc95f57f56d88341"
Nov 02 14:45:09.646610 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.646588    5034 scope.go:122] "RemoveContainer" containerID="9abd75a0a3064dfd7b6c34ffe3672104b64c83146bf82a695730db912c3c048c"
Nov 02 14:45:09.665991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.665973    5034 scope.go:122] "RemoveContainer" containerID="315efed037f7e405ee61d7eaf1985597062f47029c9e39db29f827e29a583abe"
Nov 02 14:45:09.826237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.826205    5034 scope.go:122] "RemoveContainer" containerID="b81d119011cf51f51cc04c677154c8ccb5565f41c74ae32150f57561e060c27e"
Nov 02 14:45:09.826737 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.826711    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn_odh-ai-gateway-infra(a29378f6-902d-4efb-977a-dd1f13dd5248)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248"
Nov 02 14:45:09.827960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.827932    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" event={"ID":"c63a0f7e-2b21-46b1-80d2-61d439daf047","Type":"ContainerStarted","Data":"7a2ff66c32af1b2fa662a01cdc7d2131c447367fa7e8d6d8cad0203c7456a0a0"}
Nov 02 14:45:09.830626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.830587    5034 scope.go:122] "RemoveContainer" containerID="ddb44fe7265e7fa3dc61f5dc6574973023420a8309e992eddd4732d08bc45bdc"
Nov 02 14:45:09.831073 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.831045    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn_odh-ai-gateway-infra(df67ed0b-ff8d-4774-a628-3acbf17eb2db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn" podUID="df67ed0b-ff8d-4774-a628-3acbf17eb2db"
Nov 02 14:45:09.833260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.833230    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" event={"ID":"6518f1f5-1243-49f7-8bf2-b174142e754a","Type":"ContainerStarted","Data":"56b01bc170f1d69346fc26d3e5b670d6a80db4c6538aa228c1f28878303334ed"}
Nov 02 14:45:09.835198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.835179    5034 scope.go:122] "RemoveContainer" containerID="6657a641f4f7a5482ee8906da993445b6c25769fec607b7fa2a620427031138f"
Nov 02 14:45:09.835558 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.835531    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt_odh-ai-gateway-infra(ac656679-7557-495a-a88e-8b166be2da0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" podUID="ac656679-7557-495a-a88e-8b166be2da0e"
Nov 02 14:45:09.837138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.837115    5034 generic.go:356] "Generic (PLEG): container finished" podID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerID="fa0819fba1215530776f58a8a21fbddc2e67a01804957d7c1c2650f59beffaf1" exitCode=60
Nov 02 14:45:09.837236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.837196    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" event={"ID":"3238cdd3-2e70-456b-afe8-6cdbf2b20b89","Type":"ContainerDied","Data":"fa0819fba1215530776f58a8a21fbddc2e67a01804957d7c1c2650f59beffaf1"}
Nov 02 14:45:09.837295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.837254    5034 scope.go:122] "RemoveContainer" containerID="43fa0641e208992d15a139ad17495567b476cb91d08c701703077f7a1fcf4a1c"
Nov 02 14:45:09.837445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.837426    5034 scope.go:122] "RemoveContainer" containerID="fa0819fba1215530776f58a8a21fbddc2e67a01804957d7c1c2650f59beffaf1"
Nov 02 14:45:09.837771 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.837734    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s_odh-ai-gateway-infra(3238cdd3-2e70-456b-afe8-6cdbf2b20b89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89"
Nov 02 14:45:09.839489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.839464    5034 generic.go:356] "Generic (PLEG): container finished" podID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerID="476ec33a22a342d6905570e705e2010e8b9d54a3a1db9203fe8f4543c1884edb" exitCode=60
Nov 02 14:45:09.839587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.839519    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" event={"ID":"b407ad8b-5856-4f5f-baf7-baebc7fe9233","Type":"ContainerDied","Data":"476ec33a22a342d6905570e705e2010e8b9d54a3a1db9203fe8f4543c1884edb"}
Nov 02 14:45:09.839835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.839770    5034 scope.go:122] "RemoveContainer" containerID="476ec33a22a342d6905570e705e2010e8b9d54a3a1db9203fe8f4543c1884edb"
Nov 02 14:45:09.840125 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.840103    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w_odh-ai-gateway-infra(b407ad8b-5856-4f5f-baf7-baebc7fe9233)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233"
Nov 02 14:45:09.841642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.841623    5034 scope.go:122] "RemoveContainer" containerID="c78547df16d4f9c4c2eb79e48e2a9e07c62f0860c13b02ffa1f31a934ef286ba"
Nov 02 14:45:09.841968 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.841946    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh_odh-ai-gateway-infra(82af8d05-2c41-441f-b905-a1b186bfcd11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh" podUID="82af8d05-2c41-441f-b905-a1b186bfcd11"
Nov 02 14:45:09.843462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.843442    5034 scope.go:122] "RemoveContainer" containerID="ad2bdef45e40871ec0f623c6390af9c9175e021658ccdf4995ca09a0f8f852f3"
Nov 02 14:45:09.843731 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.843709    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr_odh-ai-gateway-infra(6d83cf56-9c22-47e5-aef4-087b779a2f65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65"
Nov 02 14:45:09.847241 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.847223    5034 scope.go:122] "RemoveContainer" containerID="609b428e1321080ba0125383f79779ce35ff073ad567c7372983ac935434201f"
Nov 02 14:45:09.847539 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.847513    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj_odh-ai-gateway-infra(5b3eed1b-a630-4ee7-b3fc-de9d967c544d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d"
Nov 02 14:45:09.849277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.849256    5034 scope.go:122] "RemoveContainer" containerID="8d47d5cd66be15c1eb5f773696d0863e3de5cb5d0276f054ab20dc8a6dc71808"
Nov 02 14:45:09.849600 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.849564    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb_odh-ai-gateway-infra(5527c983-4d3e-4d15-8118-3f4f656b8051)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051"
Nov 02 14:45:09.850787 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.850758    5034 scope.go:122] "RemoveContainer" containerID="21b1155736f2f97d9206a0cb056998e08fe770a233a2cb4b5bafc23e33654456"
Nov 02 14:45:09.851046 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.851027    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq_odh-ai-gateway-infra(98bfda5e-d485-4a81-835b-4c369b3666e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4"
Nov 02 14:45:09.852533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.852516    5034 scope.go:122] "RemoveContainer" containerID="3c1a727b903e55fef6636bc699c1d354af0e3903cf2a120b78015d6e049a41b6"
Nov 02 14:45:09.852792 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.852755    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2_odh-ai-gateway-infra(c7ee5b3f-068d-40b7-aaf1-461643ff7fe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7"
Nov 02 14:45:09.854261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.854241    5034 scope.go:122] "RemoveContainer" containerID="d0d0c1abafeb962fb562ca120517b253528fd967d06fd94889b79e38b8afb8f8"
Nov 02 14:45:09.854531 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.854509    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn_odh-ai-gateway-infra(5bdbb368-93f7-4f30-89ef-881ec3528d6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c"
Nov 02 14:45:09.855721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.855702    5034 scope.go:122] "RemoveContainer" containerID="9c50143eea0404138b96059ed0586caf623c0ca500214f29f66d64092460d09e"
Nov 02 14:45:09.857792 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.857763    5034 scope.go:122] "RemoveContainer" containerID="d0385e342fedf82711eda835ecf557bcc3abbe42e0ef244e5c471edde568d51f"
Nov 02 14:45:09.858718 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.858674    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc_odh-ai-gateway-infra(708b38e0-5f4a-4669-84e7-f74b02a3425e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e"
Nov 02 14:45:09.878296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.878258    5034 scope.go:122] "RemoveContainer" containerID="a4b81ca7474705a5a7ae8c8ef357d04849c2ae94c5cd7beaaf656e95c17fe748"
Nov 02 14:45:09.879177 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.879145    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv_odh-ai-gateway-infra(0e9989de-be08-4242-944b-9db03854c4ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" podUID="0e9989de-be08-4242-944b-9db03854c4ed"
Nov 02 14:45:09.884594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.884555    5034 generic.go:356] "Generic (PLEG): container finished" podID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerID="e72a369c122d8ede1688d22861b4d15a19d55c9f516f9e64848208ac0d3f7d5f" exitCode=60
Nov 02 14:45:09.884771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.884671    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" event={"ID":"86a9e4f4-79f9-4e98-be5e-f871f11b183d","Type":"ContainerDied","Data":"e72a369c122d8ede1688d22861b4d15a19d55c9f516f9e64848208ac0d3f7d5f"}
Nov 02 14:45:09.889341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.885866    5034 scope.go:122] "RemoveContainer" containerID="e72a369c122d8ede1688d22861b4d15a19d55c9f516f9e64848208ac0d3f7d5f"
Nov 02 14:45:09.889341 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.886291    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893845-cgqjf_odh-ai-gateway-infra(86a9e4f4-79f9-4e98-be5e-f871f11b183d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d"
Nov 02 14:45:09.889341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.889283    5034 generic.go:356] "Generic (PLEG): container finished" podID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerID="69d49dc46e78620954bae32579243fb930d5501080e62acf8ed1dd0428b8a19c" exitCode=60
Nov 02 14:45:09.889769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.889357    5034 scope.go:122] "RemoveContainer" containerID="6744b80269c9c0d980a63f90a0d66bbaa90cd188ed80ff7b57f9618845aa5e02"
Nov 02 14:45:09.889769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.889430    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" event={"ID":"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f","Type":"ContainerDied","Data":"69d49dc46e78620954bae32579243fb930d5501080e62acf8ed1dd0428b8a19c"}
Nov 02 14:45:09.889935 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.889866    5034 scope.go:122] "RemoveContainer" containerID="69d49dc46e78620954bae32579243fb930d5501080e62acf8ed1dd0428b8a19c"
Nov 02 14:45:09.890324 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.890292    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk_odh-ai-gateway-infra(9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f"
Nov 02 14:45:09.893039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.892993    5034 generic.go:356] "Generic (PLEG): container finished" podID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerID="cf7660caa42c704c5fb88b9d27cfccd0b18bc63676346559d5065d0ee77331d2" exitCode=60
Nov 02 14:45:09.893168 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.893129    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" event={"ID":"4dcb0f45-1ec6-43ac-a9af-571593753a63","Type":"ContainerDied","Data":"cf7660caa42c704c5fb88b9d27cfccd0b18bc63676346559d5065d0ee77331d2"}
Nov 02 14:45:09.893458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.893434    5034 scope.go:122] "RemoveContainer" containerID="cf7660caa42c704c5fb88b9d27cfccd0b18bc63676346559d5065d0ee77331d2"
Nov 02 14:45:09.893864 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:09.893839    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn_odh-ai-gateway-infra(4dcb0f45-1ec6-43ac-a9af-571593753a63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63"
Nov 02 14:45:09.901191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.901105    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"]
Nov 02 14:45:09.902401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.901353    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy" containerID="cri-o://6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2" gracePeriod=30
Nov 02 14:45:09.916379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.914614    5034 scope.go:122] "RemoveContainer" containerID="53f6e01c2fa1bc68a11db9c6f754a6621bdfb4fd76f98eb72ebe0bcb4faa5735"
Nov 02 14:45:09.940962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:09.940935    5034 scope.go:122] "RemoveContainer" containerID="3f1a54fb5792b73da78af4723a8c9aa135b8cb3dc05d038565cfb9ca1b9ea75d"
Nov 02 14:45:10.907748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.907713    5034 generic.go:356] "Generic (PLEG): container finished" podID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerID="7a2ff66c32af1b2fa662a01cdc7d2131c447367fa7e8d6d8cad0203c7456a0a0" exitCode=60
Nov 02 14:45:10.908262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.907816    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" event={"ID":"c63a0f7e-2b21-46b1-80d2-61d439daf047","Type":"ContainerDied","Data":"7a2ff66c32af1b2fa662a01cdc7d2131c447367fa7e8d6d8cad0203c7456a0a0"}
Nov 02 14:45:10.908262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.907907    5034 scope.go:122] "RemoveContainer" containerID="7ee09109e5e0e806c7bcb099689906e67138c08cdb67518015df870561104e36"
Nov 02 14:45:10.908262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.908169    5034 scope.go:122] "RemoveContainer" containerID="7a2ff66c32af1b2fa662a01cdc7d2131c447367fa7e8d6d8cad0203c7456a0a0"
Nov 02 14:45:10.908569 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:10.908546    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x_odh-ai-gateway-infra(c63a0f7e-2b21-46b1-80d2-61d439daf047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047"
Nov 02 14:45:10.910352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.910329    5034 generic.go:356] "Generic (PLEG): container finished" podID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerID="56b01bc170f1d69346fc26d3e5b670d6a80db4c6538aa228c1f28878303334ed" exitCode=60
Nov 02 14:45:10.910443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.910414    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" event={"ID":"6518f1f5-1243-49f7-8bf2-b174142e754a","Type":"ContainerDied","Data":"56b01bc170f1d69346fc26d3e5b670d6a80db4c6538aa228c1f28878303334ed"}
Nov 02 14:45:10.910811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.910760    5034 scope.go:122] "RemoveContainer" containerID="56b01bc170f1d69346fc26d3e5b670d6a80db4c6538aa228c1f28878303334ed"
Nov 02 14:45:10.911195 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:10.911173    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m_odh-ai-gateway-infra(6518f1f5-1243-49f7-8bf2-b174142e754a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a"
Nov 02 14:45:10.912693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.912673    5034 scope.go:122] "RemoveContainer" containerID="fa0819fba1215530776f58a8a21fbddc2e67a01804957d7c1c2650f59beffaf1"
Nov 02 14:45:10.913047 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:10.913020    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s_odh-ai-gateway-infra(3238cdd3-2e70-456b-afe8-6cdbf2b20b89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89"
Nov 02 14:45:10.914411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.914390    5034 scope.go:122] "RemoveContainer" containerID="476ec33a22a342d6905570e705e2010e8b9d54a3a1db9203fe8f4543c1884edb"
Nov 02 14:45:10.914760 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:10.914731    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w_odh-ai-gateway-infra(b407ad8b-5856-4f5f-baf7-baebc7fe9233)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233"
Nov 02 14:45:10.916409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.916389    5034 scope.go:122] "RemoveContainer" containerID="e72a369c122d8ede1688d22861b4d15a19d55c9f516f9e64848208ac0d3f7d5f"
Nov 02 14:45:10.916791 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:10.916692    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893845-cgqjf_odh-ai-gateway-infra(86a9e4f4-79f9-4e98-be5e-f871f11b183d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d"
Nov 02 14:45:10.918644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.918621    5034 scope.go:122] "RemoveContainer" containerID="69d49dc46e78620954bae32579243fb930d5501080e62acf8ed1dd0428b8a19c"
Nov 02 14:45:10.918998 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:10.918970    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk_odh-ai-gateway-infra(9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f"
Nov 02 14:45:10.921375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.921357    5034 scope.go:122] "RemoveContainer" containerID="cf7660caa42c704c5fb88b9d27cfccd0b18bc63676346559d5065d0ee77331d2"
Nov 02 14:45:10.921628 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:10.921610    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn_odh-ai-gateway-infra(4dcb0f45-1ec6-43ac-a9af-571593753a63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63"
Nov 02 14:45:10.926345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:10.926328    5034 scope.go:122] "RemoveContainer" containerID="98c3142d6a6a9c34e0a009e7923b24d71318ddccb9b2b42c9a906f709fa09013"
Nov 02 14:45:11.934691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:11.934658    5034 scope.go:122] "RemoveContainer" containerID="7a2ff66c32af1b2fa662a01cdc7d2131c447367fa7e8d6d8cad0203c7456a0a0"
Nov 02 14:45:11.935210 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:11.935148    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x_odh-ai-gateway-infra(c63a0f7e-2b21-46b1-80d2-61d439daf047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047"
Nov 02 14:45:11.936656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:11.936634    5034 scope.go:122] "RemoveContainer" containerID="56b01bc170f1d69346fc26d3e5b670d6a80db4c6538aa228c1f28878303334ed"
Nov 02 14:45:11.937014 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:11.936990    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m_odh-ai-gateway-infra(6518f1f5-1243-49f7-8bf2-b174142e754a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a"
Nov 02 14:45:13.232222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:13.232185    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz" podUID="de00d3b1-f2d5-46c7-a29c-fb1576200196" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:45:13.998915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:13.998750    5034 generic.go:356] "Generic (PLEG): container finished" podID="de00d3b1-f2d5-46c7-a29c-fb1576200196" containerID="fd59f174a079fc1377c4ad159c94a7acb9899cea2e88cc5ddf51fa84d4b6b6f2" exitCode=0
Nov 02 14:45:13.999172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:13.999141    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz" event={"ID":"de00d3b1-f2d5-46c7-a29c-fb1576200196","Type":"ContainerDied","Data":"fd59f174a079fc1377c4ad159c94a7acb9899cea2e88cc5ddf51fa84d4b6b6f2"}
Nov 02 14:45:14.095873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.095839    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:45:14.184108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.184074    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de00d3b1-f2d5-46c7-a29c-fb1576200196-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de00d3b1-f2d5-46c7-a29c-fb1576200196-plugins-config-volume\") pod \"de00d3b1-f2d5-46c7-a29c-fb1576200196\" (UID: \"de00d3b1-f2d5-46c7-a29c-fb1576200196\") "
Nov 02 14:45:14.184512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.184483    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de00d3b1-f2d5-46c7-a29c-fb1576200196-plugins-config-volume" pod "de00d3b1-f2d5-46c7-a29c-fb1576200196" (UID: "de00d3b1-f2d5-46c7-a29c-fb1576200196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:14.184609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.184540    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de00d3b1-f2d5-46c7-a29c-fb1576200196-kube-api-access-7j4xf\" (UniqueName: \"kubernetes.io/projected/de00d3b1-f2d5-46c7-a29c-fb1576200196-kube-api-access-7j4xf\") pod \"de00d3b1-f2d5-46c7-a29c-fb1576200196\" (UID: \"de00d3b1-f2d5-46c7-a29c-fb1576200196\") "
Nov 02 14:45:14.185312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.185284    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de00d3b1-f2d5-46c7-a29c-fb1576200196-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:14.187255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.187218    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de00d3b1-f2d5-46c7-a29c-fb1576200196-kube-api-access-7j4xf" pod "de00d3b1-f2d5-46c7-a29c-fb1576200196" (UID: "de00d3b1-f2d5-46c7-a29c-fb1576200196"). InnerVolumeSpecName "kube-api-access-7j4xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:14.287699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.287661    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j4xf\" (UniqueName: \"kubernetes.io/projected/de00d3b1-f2d5-46c7-a29c-fb1576200196-kube-api-access-7j4xf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:14.359332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.359233    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"]
Nov 02 14:45:14.658891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.651564    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:14.701801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.699879    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82af8d05-2c41-441f-b905-a1b186bfcd11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82af8d05-2c41-441f-b905-a1b186bfcd11-maas-api-service-ca\") pod \"82af8d05-2c41-441f-b905-a1b186bfcd11\" (UID: \"82af8d05-2c41-441f-b905-a1b186bfcd11\") "
Nov 02 14:45:14.701801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.699943    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82af8d05-2c41-441f-b905-a1b186bfcd11-kube-api-access-l5crt\" (UniqueName: \"kubernetes.io/projected/82af8d05-2c41-441f-b905-a1b186bfcd11-kube-api-access-l5crt\") pod \"82af8d05-2c41-441f-b905-a1b186bfcd11\" (UID: \"82af8d05-2c41-441f-b905-a1b186bfcd11\") "
Nov 02 14:45:14.702113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.702076    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82af8d05-2c41-441f-b905-a1b186bfcd11-maas-api-service-ca" pod "82af8d05-2c41-441f-b905-a1b186bfcd11" (UID: "82af8d05-2c41-441f-b905-a1b186bfcd11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:14.710905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.710537    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82af8d05-2c41-441f-b905-a1b186bfcd11-kube-api-access-l5crt" pod "82af8d05-2c41-441f-b905-a1b186bfcd11" (UID: "82af8d05-2c41-441f-b905-a1b186bfcd11"). InnerVolumeSpecName "kube-api-access-l5crt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:14.834802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.830970    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82af8d05-2c41-441f-b905-a1b186bfcd11-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:14.834802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:14.831006    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5crt\" (UniqueName: \"kubernetes.io/projected/82af8d05-2c41-441f-b905-a1b186bfcd11-kube-api-access-l5crt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.138253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.138209    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"]
Nov 02 14:45:15.451122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.451068    5034 scope.go:122] "RemoveContainer" containerID="45cfa36a84f7ee6ff73352166935312eeee9f7542dbac1f1b709011d9975b20f"
Nov 02 14:45:15.511838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.511012    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"
Nov 02 14:45:15.519404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.519365    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz" event={"ID":"de00d3b1-f2d5-46c7-a29c-fb1576200196","Type":"ContainerDied","Data":"2e4bb5e8dc3c72c2f89047e4b7b067c5ac26930b588eb728a23b8cd2c55fc7c8"}
Nov 02 14:45:15.536002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.534549    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh" event={"ID":"82af8d05-2c41-441f-b905-a1b186bfcd11","Type":"ContainerDied","Data":"93a6a5367393272affd047d1e16e0993c1678101d7ab59f6d27dcc2cb684f313"}
Nov 02 14:45:15.536002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.534615    5034 scope.go:122] "RemoveContainer" containerID="fd59f174a079fc1377c4ad159c94a7acb9899cea2e88cc5ddf51fa84d4b6b6f2"
Nov 02 14:45:15.536002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.527086    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"
Nov 02 14:45:15.620200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.620168    5034 scope.go:122] "RemoveContainer" containerID="c78547df16d4f9c4c2eb79e48e2a9e07c62f0860c13b02ffa1f31a934ef286ba"
Nov 02 14:45:15.630383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.630029    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"]
Nov 02 14:45:15.638998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.638920    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e1fcb4-74685db74c-dbclz"]
Nov 02 14:45:15.686460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.686378    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"]
Nov 02 14:45:15.695215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.695151    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e1fcb4-29893845-qwsrh"]
Nov 02 14:45:15.854219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.854187    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:15.857841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.857770    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:45:15.956700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.956608    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-kube-api-access-j9vsn\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-kube-api-access-j9vsn\") pod \"28b44b24-e712-4d2e-9a1d-9c219925ff44\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") "
Nov 02 14:45:15.956895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.956720    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-certs\") pod \"28b44b24-e712-4d2e-9a1d-9c219925ff44\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") "
Nov 02 14:45:15.956895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.956857    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-envoy\") pod \"28b44b24-e712-4d2e-9a1d-9c219925ff44\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") "
Nov 02 14:45:15.957039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.956931    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b44b24-e712-4d2e-9a1d-9c219925ff44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b44b24-e712-4d2e-9a1d-9c219925ff44-istiod-ca-cert\") pod \"28b44b24-e712-4d2e-9a1d-9c219925ff44\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") "
Nov 02 14:45:15.957039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.956962    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-token\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-token\") pod \"28b44b24-e712-4d2e-9a1d-9c219925ff44\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") "
Nov 02 14:45:15.957149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.957134    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-data\") pod \"28b44b24-e712-4d2e-9a1d-9c219925ff44\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") "
Nov 02 14:45:15.957263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.957239    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df67ed0b-ff8d-4774-a628-3acbf17eb2db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df67ed0b-ff8d-4774-a628-3acbf17eb2db-maas-api-service-ca\") pod \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\" (UID: \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\") "
Nov 02 14:45:15.957412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.957279    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-credential-socket\") pod \"28b44b24-e712-4d2e-9a1d-9c219925ff44\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") "
Nov 02 14:45:15.957412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.957371    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df67ed0b-ff8d-4774-a628-3acbf17eb2db-kube-api-access-hmqhp\" (UniqueName: \"kubernetes.io/projected/df67ed0b-ff8d-4774-a628-3acbf17eb2db-kube-api-access-hmqhp\") pod \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\" (UID: \"df67ed0b-ff8d-4774-a628-3acbf17eb2db\") "
Nov 02 14:45:15.957536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.957475    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-podinfo\") pod \"28b44b24-e712-4d2e-9a1d-9c219925ff44\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") "
Nov 02 14:45:15.957536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.957504    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-socket\") pod \"28b44b24-e712-4d2e-9a1d-9c219925ff44\" (UID: \"28b44b24-e712-4d2e-9a1d-9c219925ff44\") "
Nov 02 14:45:15.958666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.957900    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-credential-socket" pod "28b44b24-e712-4d2e-9a1d-9c219925ff44" (UID: "28b44b24-e712-4d2e-9a1d-9c219925ff44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:15.958666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.958107    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-certs" pod "28b44b24-e712-4d2e-9a1d-9c219925ff44" (UID: "28b44b24-e712-4d2e-9a1d-9c219925ff44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:15.958666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.958286    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-data" pod "28b44b24-e712-4d2e-9a1d-9c219925ff44" (UID: "28b44b24-e712-4d2e-9a1d-9c219925ff44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:15.958666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.958543    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-socket" pod "28b44b24-e712-4d2e-9a1d-9c219925ff44" (UID: "28b44b24-e712-4d2e-9a1d-9c219925ff44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:15.958666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.958599    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.958666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.958620    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.958666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.958634    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:15.959219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.958857    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df67ed0b-ff8d-4774-a628-3acbf17eb2db-maas-api-service-ca" pod "df67ed0b-ff8d-4774-a628-3acbf17eb2db" (UID: "df67ed0b-ff8d-4774-a628-3acbf17eb2db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:15.959483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.959448    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b44b24-e712-4d2e-9a1d-9c219925ff44-istiod-ca-cert" pod "28b44b24-e712-4d2e-9a1d-9c219925ff44" (UID: "28b44b24-e712-4d2e-9a1d-9c219925ff44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:15.962482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.962443    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-podinfo" pod "28b44b24-e712-4d2e-9a1d-9c219925ff44" (UID: "28b44b24-e712-4d2e-9a1d-9c219925ff44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:45:15.962932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.962908    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-envoy" pod "28b44b24-e712-4d2e-9a1d-9c219925ff44" (UID: "28b44b24-e712-4d2e-9a1d-9c219925ff44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:15.963432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.963401    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df67ed0b-ff8d-4774-a628-3acbf17eb2db-kube-api-access-hmqhp" pod "df67ed0b-ff8d-4774-a628-3acbf17eb2db" (UID: "df67ed0b-ff8d-4774-a628-3acbf17eb2db"). InnerVolumeSpecName "kube-api-access-hmqhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:15.963995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.963974    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-token" pod "28b44b24-e712-4d2e-9a1d-9c219925ff44" (UID: "28b44b24-e712-4d2e-9a1d-9c219925ff44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:15.965159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:15.965130    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-kube-api-access-j9vsn" pod "28b44b24-e712-4d2e-9a1d-9c219925ff44" (UID: "28b44b24-e712-4d2e-9a1d-9c219925ff44"). InnerVolumeSpecName "kube-api-access-j9vsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:16.061952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.061910    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9vsn\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-kube-api-access-j9vsn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:16.061952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.061954    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:16.062244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.061968    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b44b24-e712-4d2e-9a1d-9c219925ff44-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:16.062244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.061980    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:16.062244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.061993    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df67ed0b-ff8d-4774-a628-3acbf17eb2db-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:16.062244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.062005    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmqhp\" (UniqueName: \"kubernetes.io/projected/df67ed0b-ff8d-4774-a628-3acbf17eb2db-kube-api-access-hmqhp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:16.062244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.062022    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b44b24-e712-4d2e-9a1d-9c219925ff44-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:16.062244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.062035    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b44b24-e712-4d2e-9a1d-9c219925ff44-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:16.538184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.538150    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"
Nov 02 14:45:16.538184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.538167    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn" event={"ID":"df67ed0b-ff8d-4774-a628-3acbf17eb2db","Type":"ContainerDied","Data":"9ec144dcc01d9da9ea0835ffe4dd58273fdf5af22a1301332cd35d9ac110d898"}
Nov 02 14:45:16.538710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.538231    5034 scope.go:122] "RemoveContainer" containerID="ddb44fe7265e7fa3dc61f5dc6574973023420a8309e992eddd4732d08bc45bdc"
Nov 02 14:45:16.543599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.543568    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" event={"ID":"cd77d21c-5627-41d7-ab81-ec7a4651f524","Type":"ContainerStarted","Data":"80cf9787c14130ee0208157d694256d5b7106c8ccc56f033a5ea3edc972a2e73"}
Nov 02 14:45:16.545388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.545358    5034 generic.go:356] "Generic (PLEG): container finished" podID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerID="6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2" exitCode=0
Nov 02 14:45:16.545515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.545433    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" event={"ID":"28b44b24-e712-4d2e-9a1d-9c219925ff44","Type":"ContainerDied","Data":"6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2"}
Nov 02 14:45:16.545515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.545483    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm" event={"ID":"28b44b24-e712-4d2e-9a1d-9c219925ff44","Type":"ContainerDied","Data":"5f56659685e4b2106a4358961ccd7e158bda11d205fc956ee1e867b6efb36dc9"}
Nov 02 14:45:16.545634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.545527    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"
Nov 02 14:45:16.554494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.554459    5034 scope.go:122] "RemoveContainer" containerID="6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2"
Nov 02 14:45:16.572194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.572162    5034 scope.go:122] "RemoveContainer" containerID="6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2"
Nov 02 14:45:16.572589 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:16.572560    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2\": container with ID starting with 6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2 not found: ID does not exist" containerID="6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2"
Nov 02 14:45:16.572722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.572600    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2"} err="failed to get container status \"6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2\": rpc error: code = NotFound desc = could not find container \"6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2\": container with ID starting with 6efa84d683ae2121fd3bb2c54eefb20c27ffaeb571a9430032d8b8c93ad017f2 not found: ID does not exist"
Nov 02 14:45:16.622861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.622803    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" podStartSLOduration=16.622787531 podStartE2EDuration="16.622787531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:16.617137695 +0000 UTC m=+3707.368442133" watchObservedRunningTime="2026-11-02 14:45:16.622787531 +0000 UTC m=+3707.374091959"
Nov 02 14:45:16.664327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.664288    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"]
Nov 02 14:45:16.670433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.670393    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-09d1a5-29893845-xq5tn"]
Nov 02 14:45:16.712111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.712079    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"]
Nov 02 14:45:16.718201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:16.718171    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-qkbsm"]
Nov 02 14:45:17.399342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.399298    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" path="/var/lib/kubelet/pods/28b44b24-e712-4d2e-9a1d-9c219925ff44/volumes"
Nov 02 14:45:17.399928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.399906    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82af8d05-2c41-441f-b905-a1b186bfcd11" path="/var/lib/kubelet/pods/82af8d05-2c41-441f-b905-a1b186bfcd11/volumes"
Nov 02 14:45:17.400410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.400390    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de00d3b1-f2d5-46c7-a29c-fb1576200196" path="/var/lib/kubelet/pods/de00d3b1-f2d5-46c7-a29c-fb1576200196/volumes"
Nov 02 14:45:17.401198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.401178    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df67ed0b-ff8d-4774-a628-3acbf17eb2db" path="/var/lib/kubelet/pods/df67ed0b-ff8d-4774-a628-3acbf17eb2db/volumes"
Nov 02 14:45:17.560197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.560159    5034 generic.go:356] "Generic (PLEG): container finished" podID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerID="80cf9787c14130ee0208157d694256d5b7106c8ccc56f033a5ea3edc972a2e73" exitCode=60
Nov 02 14:45:17.560646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.560227    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" event={"ID":"cd77d21c-5627-41d7-ab81-ec7a4651f524","Type":"ContainerDied","Data":"80cf9787c14130ee0208157d694256d5b7106c8ccc56f033a5ea3edc972a2e73"}
Nov 02 14:45:17.560646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.560289    5034 scope.go:122] "RemoveContainer" containerID="45cfa36a84f7ee6ff73352166935312eeee9f7542dbac1f1b709011d9975b20f"
Nov 02 14:45:17.560851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.560830    5034 scope.go:122] "RemoveContainer" containerID="80cf9787c14130ee0208157d694256d5b7106c8ccc56f033a5ea3edc972a2e73"
Nov 02 14:45:17.561643 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:17.561338    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp_odh-ai-gateway-infra(cd77d21c-5627-41d7-ab81-ec7a4651f524)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524"
Nov 02 14:45:17.630839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.630802    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"]
Nov 02 14:45:17.968456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.968431    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"]
Nov 02 14:45:17.971850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:17.968862    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy" containerID="cri-o://ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8" gracePeriod=30
Nov 02 14:45:18.418015 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:18.417963    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:45:18.784681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:18.784650    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:18.825448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:18.825413    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd77d21c-5627-41d7-ab81-ec7a4651f524-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77d21c-5627-41d7-ab81-ec7a4651f524-maas-api-service-ca\") pod \"cd77d21c-5627-41d7-ab81-ec7a4651f524\" (UID: \"cd77d21c-5627-41d7-ab81-ec7a4651f524\") "
Nov 02 14:45:18.825808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:18.825771    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd77d21c-5627-41d7-ab81-ec7a4651f524-kube-api-access-66kl2\" (UniqueName: \"kubernetes.io/projected/cd77d21c-5627-41d7-ab81-ec7a4651f524-kube-api-access-66kl2\") pod \"cd77d21c-5627-41d7-ab81-ec7a4651f524\" (UID: \"cd77d21c-5627-41d7-ab81-ec7a4651f524\") "
Nov 02 14:45:18.825986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:18.825884    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd77d21c-5627-41d7-ab81-ec7a4651f524-maas-api-service-ca" pod "cd77d21c-5627-41d7-ab81-ec7a4651f524" (UID: "cd77d21c-5627-41d7-ab81-ec7a4651f524"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:18.826494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:18.826469    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd77d21c-5627-41d7-ab81-ec7a4651f524-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:18.828361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:18.828330    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd77d21c-5627-41d7-ab81-ec7a4651f524-kube-api-access-66kl2" pod "cd77d21c-5627-41d7-ab81-ec7a4651f524" (UID: "cd77d21c-5627-41d7-ab81-ec7a4651f524"). InnerVolumeSpecName "kube-api-access-66kl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:18.883726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:18.883684    5034 patch_prober.go:29] interesting pod/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:45:18.883938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:18.883743    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:45:18.928121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:18.928075    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66kl2\" (UniqueName: \"kubernetes.io/projected/cd77d21c-5627-41d7-ab81-ec7a4651f524-kube-api-access-66kl2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:19.395510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.395467    5034 scope.go:122] "RemoveContainer" containerID="d0385e342fedf82711eda835ecf557bcc3abbe42e0ef244e5c471edde568d51f"
Nov 02 14:45:19.396761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.396455    5034 scope.go:122] "RemoveContainer" containerID="609b428e1321080ba0125383f79779ce35ff073ad567c7372983ac935434201f"
Nov 02 14:45:19.397124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.397099    5034 scope.go:122] "RemoveContainer" containerID="0af883b6767615d6bab52a43bbb6b84848945b3dc4bbac77303ea777eb070f9c"
Nov 02 14:45:19.397578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.397558    5034 scope.go:122] "RemoveContainer" containerID="6657a641f4f7a5482ee8906da993445b6c25769fec607b7fa2a620427031138f"
Nov 02 14:45:19.399052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.399030    5034 scope.go:122] "RemoveContainer" containerID="21b1155736f2f97d9206a0cb056998e08fe770a233a2cb4b5bafc23e33654456"
Nov 02 14:45:19.585012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.584978    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"
Nov 02 14:45:19.585219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.585004    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp" event={"ID":"cd77d21c-5627-41d7-ab81-ec7a4651f524","Type":"ContainerDied","Data":"15dabd8cd07dbbeb9e4d5587ec943e743055ba1e4268ff96c04ac03617ebab35"}
Nov 02 14:45:19.585219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.585080    5034 scope.go:122] "RemoveContainer" containerID="80cf9787c14130ee0208157d694256d5b7106c8ccc56f033a5ea3edc972a2e73"
Nov 02 14:45:19.648207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.648119    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"]
Nov 02 14:45:19.661414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:19.661380    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fe1881-29893845-tjdzp"]
Nov 02 14:45:20.387698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.387592    5034 scope.go:122] "RemoveContainer" containerID="cf7660caa42c704c5fb88b9d27cfccd0b18bc63676346559d5065d0ee77331d2"
Nov 02 14:45:20.388190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.387847    5034 scope.go:122] "RemoveContainer" containerID="a4b81ca7474705a5a7ae8c8ef357d04849c2ae94c5cd7beaaf656e95c17fe748"
Nov 02 14:45:20.388190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.388058    5034 scope.go:122] "RemoveContainer" containerID="69d49dc46e78620954bae32579243fb930d5501080e62acf8ed1dd0428b8a19c"
Nov 02 14:45:20.389297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.388505    5034 scope.go:122] "RemoveContainer" containerID="ad2bdef45e40871ec0f623c6390af9c9175e021658ccdf4995ca09a0f8f852f3"
Nov 02 14:45:20.389297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.388619    5034 scope.go:122] "RemoveContainer" containerID="476ec33a22a342d6905570e705e2010e8b9d54a3a1db9203fe8f4543c1884edb"
Nov 02 14:45:20.599296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.599257    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" event={"ID":"7b296f5f-1030-4705-998d-4459d0706cb0","Type":"ContainerStarted","Data":"68d291693cf03d37186bd90f9e260ff6b3c01646bca11080ea0ff6408ac8b73f"}
Nov 02 14:45:20.601509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.601477    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" event={"ID":"5b3eed1b-a630-4ee7-b3fc-de9d967c544d","Type":"ContainerStarted","Data":"5af0d06cade01907d37c219fecacdb2f92f9e2f3bd8f11aff7bedd221163c8cc"}
Nov 02 14:45:20.603721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.603693    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" event={"ID":"98bfda5e-d485-4a81-835b-4c369b3666e4","Type":"ContainerStarted","Data":"aec0492ad3bb5189eb40a860592824cde9841281193d43f3471e0bf322b3a460"}
Nov 02 14:45:20.605960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.605935    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" event={"ID":"708b38e0-5f4a-4669-84e7-f74b02a3425e","Type":"ContainerStarted","Data":"1a963406a97f5b897636e9289cfd6c4b8a62cb887e2dbb6fecc3f41db734a81b"}
Nov 02 14:45:20.608455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.608429    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" event={"ID":"ac656679-7557-495a-a88e-8b166be2da0e","Type":"ContainerStarted","Data":"d20919307627d31956761183b73e04f782333b46a9c323320b6c715479ff7e41"}
Nov 02 14:45:20.689652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.689584    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" podStartSLOduration=20.689563361 podStartE2EDuration="20.689563361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:20.687925692 +0000 UTC m=+3711.439230129" watchObservedRunningTime="2026-11-02 14:45:20.689563361 +0000 UTC m=+3711.440867800"
Nov 02 14:45:20.777093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:20.777026    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" podStartSLOduration=20.777003598 podStartE2EDuration="20.777003598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:20.774731928 +0000 UTC m=+3711.526036369" watchObservedRunningTime="2026-11-02 14:45:20.777003598 +0000 UTC m=+3711.528308037"
Nov 02 14:45:21.388500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.387900    5034 scope.go:122] "RemoveContainer" containerID="8d47d5cd66be15c1eb5f773696d0863e3de5cb5d0276f054ab20dc8a6dc71808"
Nov 02 14:45:21.388500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.388480    5034 scope.go:122] "RemoveContainer" containerID="3da29a98819607edd645993da30b78da4d5839d1f56af227c9e88d35e38c877c"
Nov 02 14:45:21.389071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.388616    5034 scope.go:122] "RemoveContainer" containerID="b81d119011cf51f51cc04c677154c8ccb5565f41c74ae32150f57561e060c27e"
Nov 02 14:45:21.389071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.388837    5034 scope.go:122] "RemoveContainer" containerID="fa0819fba1215530776f58a8a21fbddc2e67a01804957d7c1c2650f59beffaf1"
Nov 02 14:45:21.410191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.410154    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" path="/var/lib/kubelet/pods/cd77d21c-5627-41d7-ab81-ec7a4651f524/volumes"
Nov 02 14:45:21.621712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.621675    5034 generic.go:356] "Generic (PLEG): container finished" podID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerID="1a963406a97f5b897636e9289cfd6c4b8a62cb887e2dbb6fecc3f41db734a81b" exitCode=60
Nov 02 14:45:21.621910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.621747    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" event={"ID":"708b38e0-5f4a-4669-84e7-f74b02a3425e","Type":"ContainerDied","Data":"1a963406a97f5b897636e9289cfd6c4b8a62cb887e2dbb6fecc3f41db734a81b"}
Nov 02 14:45:21.621910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.621835    5034 scope.go:122] "RemoveContainer" containerID="d0385e342fedf82711eda835ecf557bcc3abbe42e0ef244e5c471edde568d51f"
Nov 02 14:45:21.622431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.622402    5034 scope.go:122] "RemoveContainer" containerID="1a963406a97f5b897636e9289cfd6c4b8a62cb887e2dbb6fecc3f41db734a81b"
Nov 02 14:45:21.622965 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:21.622911    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc_odh-ai-gateway-infra(708b38e0-5f4a-4669-84e7-f74b02a3425e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e"
Nov 02 14:45:21.624131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.624107    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" event={"ID":"0e9989de-be08-4242-944b-9db03854c4ed","Type":"ContainerStarted","Data":"de9d916d6518974f358f058b63de280845ae8e6d62a63a08aaebb1e328034472"}
Nov 02 14:45:21.626201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.626179    5034 generic.go:356] "Generic (PLEG): container finished" podID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerID="1493a09ac19b138e2cc93dd3cb1ea8c5ab048f5cd21cadd254da1b9a43634676" exitCode=60
Nov 02 14:45:21.626304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.626266    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" event={"ID":"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f","Type":"ContainerDied","Data":"1493a09ac19b138e2cc93dd3cb1ea8c5ab048f5cd21cadd254da1b9a43634676"}
Nov 02 14:45:21.626735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.626714    5034 scope.go:122] "RemoveContainer" containerID="1493a09ac19b138e2cc93dd3cb1ea8c5ab048f5cd21cadd254da1b9a43634676"
Nov 02 14:45:21.627250 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:21.627225    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk_odh-ai-gateway-infra(9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f"
Nov 02 14:45:21.629000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.628979    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" event={"ID":"4dcb0f45-1ec6-43ac-a9af-571593753a63","Type":"ContainerStarted","Data":"1c0b6edaf53d29427fa76dae079c9072e5d622c4c56184651428dcd511a97ba0"}
Nov 02 14:45:21.631212 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.631192    5034 generic.go:356] "Generic (PLEG): container finished" podID="ac656679-7557-495a-a88e-8b166be2da0e" containerID="d20919307627d31956761183b73e04f782333b46a9c323320b6c715479ff7e41" exitCode=60
Nov 02 14:45:21.631323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.631265    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" event={"ID":"ac656679-7557-495a-a88e-8b166be2da0e","Type":"ContainerDied","Data":"d20919307627d31956761183b73e04f782333b46a9c323320b6c715479ff7e41"}
Nov 02 14:45:21.631651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.631633    5034 scope.go:122] "RemoveContainer" containerID="d20919307627d31956761183b73e04f782333b46a9c323320b6c715479ff7e41"
Nov 02 14:45:21.632005 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:21.631985    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt_odh-ai-gateway-infra(ac656679-7557-495a-a88e-8b166be2da0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" podUID="ac656679-7557-495a-a88e-8b166be2da0e"
Nov 02 14:45:21.633503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.633478    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" event={"ID":"b407ad8b-5856-4f5f-baf7-baebc7fe9233","Type":"ContainerStarted","Data":"0a826660f48a81dce3fa56a603c55a8b4d793d9a7b338b4b2a4b110f73c0ca81"}
Nov 02 14:45:21.635517 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.635495    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" event={"ID":"6d83cf56-9c22-47e5-aef4-087b779a2f65","Type":"ContainerStarted","Data":"b31619433d72dedc9e15f8b41c914792829300d4b25a955c30014b498d45b6f4"}
Nov 02 14:45:21.637576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.637557    5034 generic.go:356] "Generic (PLEG): container finished" podID="7b296f5f-1030-4705-998d-4459d0706cb0" containerID="68d291693cf03d37186bd90f9e260ff6b3c01646bca11080ea0ff6408ac8b73f" exitCode=60
Nov 02 14:45:21.637682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.637623    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" event={"ID":"7b296f5f-1030-4705-998d-4459d0706cb0","Type":"ContainerDied","Data":"68d291693cf03d37186bd90f9e260ff6b3c01646bca11080ea0ff6408ac8b73f"}
Nov 02 14:45:21.637995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.637973    5034 scope.go:122] "RemoveContainer" containerID="68d291693cf03d37186bd90f9e260ff6b3c01646bca11080ea0ff6408ac8b73f"
Nov 02 14:45:21.638275 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:21.638256    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr_odh-ai-gateway-infra(7b296f5f-1030-4705-998d-4459d0706cb0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" podUID="7b296f5f-1030-4705-998d-4459d0706cb0"
Nov 02 14:45:21.639657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.639637    5034 generic.go:356] "Generic (PLEG): container finished" podID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerID="5af0d06cade01907d37c219fecacdb2f92f9e2f3bd8f11aff7bedd221163c8cc" exitCode=60
Nov 02 14:45:21.639760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.639697    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" event={"ID":"5b3eed1b-a630-4ee7-b3fc-de9d967c544d","Type":"ContainerDied","Data":"5af0d06cade01907d37c219fecacdb2f92f9e2f3bd8f11aff7bedd221163c8cc"}
Nov 02 14:45:21.640055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.640037    5034 scope.go:122] "RemoveContainer" containerID="69d49dc46e78620954bae32579243fb930d5501080e62acf8ed1dd0428b8a19c"
Nov 02 14:45:21.640123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.640110    5034 scope.go:122] "RemoveContainer" containerID="5af0d06cade01907d37c219fecacdb2f92f9e2f3bd8f11aff7bedd221163c8cc"
Nov 02 14:45:21.640466 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:21.640443    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj_odh-ai-gateway-infra(5b3eed1b-a630-4ee7-b3fc-de9d967c544d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d"
Nov 02 14:45:21.641656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.641638    5034 generic.go:356] "Generic (PLEG): container finished" podID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerID="aec0492ad3bb5189eb40a860592824cde9841281193d43f3471e0bf322b3a460" exitCode=60
Nov 02 14:45:21.641736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.641690    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" event={"ID":"98bfda5e-d485-4a81-835b-4c369b3666e4","Type":"ContainerDied","Data":"aec0492ad3bb5189eb40a860592824cde9841281193d43f3471e0bf322b3a460"}
Nov 02 14:45:21.642033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.642017    5034 scope.go:122] "RemoveContainer" containerID="aec0492ad3bb5189eb40a860592824cde9841281193d43f3471e0bf322b3a460"
Nov 02 14:45:21.642279 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:21.642262    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq_odh-ai-gateway-infra(98bfda5e-d485-4a81-835b-4c369b3666e4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4"
Nov 02 14:45:21.655650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.655620    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"]
Nov 02 14:45:21.657272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.657254    5034 scope.go:122] "RemoveContainer" containerID="6657a641f4f7a5482ee8906da993445b6c25769fec607b7fa2a620427031138f"
Nov 02 14:45:21.689626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.689599    5034 scope.go:122] "RemoveContainer" containerID="0af883b6767615d6bab52a43bbb6b84848945b3dc4bbac77303ea777eb070f9c"
Nov 02 14:45:21.712129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.712089    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"]
Nov 02 14:45:21.736148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.736121    5034 scope.go:122] "RemoveContainer" containerID="609b428e1321080ba0125383f79779ce35ff073ad567c7372983ac935434201f"
Nov 02 14:45:21.740730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.740681    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" podStartSLOduration=21.740667775 podStartE2EDuration="21.740667775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:21.736157337 +0000 UTC m=+3712.487461778" watchObservedRunningTime="2026-11-02 14:45:21.740667775 +0000 UTC m=+3712.491972265"
Nov 02 14:45:21.753584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.753443    5034 scope.go:122] "RemoveContainer" containerID="21b1155736f2f97d9206a0cb056998e08fe770a233a2cb4b5bafc23e33654456"
Nov 02 14:45:21.769976 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.769892    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"]
Nov 02 14:45:21.833155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.833104    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"]
Nov 02 14:45:21.882508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:21.882469    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"]
Nov 02 14:45:22.387996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.387964    5034 scope.go:122] "RemoveContainer" containerID="e72a369c122d8ede1688d22861b4d15a19d55c9f516f9e64848208ac0d3f7d5f"
Nov 02 14:45:22.388355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.388333    5034 scope.go:122] "RemoveContainer" containerID="3c1a727b903e55fef6636bc699c1d354af0e3903cf2a120b78015d6e049a41b6"
Nov 02 14:45:22.655405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.655373    5034 generic.go:356] "Generic (PLEG): container finished" podID="0e9989de-be08-4242-944b-9db03854c4ed" containerID="de9d916d6518974f358f058b63de280845ae8e6d62a63a08aaebb1e328034472" exitCode=60
Nov 02 14:45:22.655839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.655453    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" event={"ID":"0e9989de-be08-4242-944b-9db03854c4ed","Type":"ContainerDied","Data":"de9d916d6518974f358f058b63de280845ae8e6d62a63a08aaebb1e328034472"}
Nov 02 14:45:22.655839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.655531    5034 scope.go:122] "RemoveContainer" containerID="a4b81ca7474705a5a7ae8c8ef357d04849c2ae94c5cd7beaaf656e95c17fe748"
Nov 02 14:45:22.656124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.656084    5034 scope.go:122] "RemoveContainer" containerID="de9d916d6518974f358f058b63de280845ae8e6d62a63a08aaebb1e328034472"
Nov 02 14:45:22.656677 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:22.656587    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv_odh-ai-gateway-infra(0e9989de-be08-4242-944b-9db03854c4ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" podUID="0e9989de-be08-4242-944b-9db03854c4ed"
Nov 02 14:45:22.666832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.666794    5034 generic.go:356] "Generic (PLEG): container finished" podID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerID="1c0b6edaf53d29427fa76dae079c9072e5d622c4c56184651428dcd511a97ba0" exitCode=60
Nov 02 14:45:22.667059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.666828    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" event={"ID":"4dcb0f45-1ec6-43ac-a9af-571593753a63","Type":"ContainerDied","Data":"1c0b6edaf53d29427fa76dae079c9072e5d622c4c56184651428dcd511a97ba0"}
Nov 02 14:45:22.667392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.667369    5034 scope.go:122] "RemoveContainer" containerID="1c0b6edaf53d29427fa76dae079c9072e5d622c4c56184651428dcd511a97ba0"
Nov 02 14:45:22.667859 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:22.667833    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn_odh-ai-gateway-infra(4dcb0f45-1ec6-43ac-a9af-571593753a63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63"
Nov 02 14:45:22.673260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.672835    5034 generic.go:356] "Generic (PLEG): container finished" podID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerID="d2c29047bd2cae7637b4623fbc40025b49718a6426fae85ce1a5845f35a26916" exitCode=60
Nov 02 14:45:22.673260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.672973    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26" event={"ID":"de34363d-7a31-4d79-8471-9b4e69d181d5","Type":"ContainerDied","Data":"d2c29047bd2cae7637b4623fbc40025b49718a6426fae85ce1a5845f35a26916"}
Nov 02 14:45:22.675756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.675147    5034 scope.go:122] "RemoveContainer" containerID="d2c29047bd2cae7637b4623fbc40025b49718a6426fae85ce1a5845f35a26916"
Nov 02 14:45:22.675756 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:22.675682    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26_odh-ai-gateway-infra(de34363d-7a31-4d79-8471-9b4e69d181d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5"
Nov 02 14:45:22.680715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.679186    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" event={"ID":"a29378f6-902d-4efb-977a-dd1f13dd5248","Type":"ContainerStarted","Data":"ae43884f817d03f487ea36aa36e47bc4d911d1a439741a168eef0cabc9b9a3e5"}
Nov 02 14:45:22.680715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.680565    5034 scope.go:122] "RemoveContainer" containerID="cf7660caa42c704c5fb88b9d27cfccd0b18bc63676346559d5065d0ee77331d2"
Nov 02 14:45:22.689751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.689294    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" event={"ID":"3238cdd3-2e70-456b-afe8-6cdbf2b20b89","Type":"ContainerStarted","Data":"95d9a1abf319246cd7ba64038e30f48fe4cbd5f9a38153e81211e563bb9eda6f"}
Nov 02 14:45:22.692117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.692078    5034 generic.go:356] "Generic (PLEG): container finished" podID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerID="0a826660f48a81dce3fa56a603c55a8b4d793d9a7b338b4b2a4b110f73c0ca81" exitCode=60
Nov 02 14:45:22.692256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.692086    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" event={"ID":"b407ad8b-5856-4f5f-baf7-baebc7fe9233","Type":"ContainerDied","Data":"0a826660f48a81dce3fa56a603c55a8b4d793d9a7b338b4b2a4b110f73c0ca81"}
Nov 02 14:45:22.692993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.692958    5034 scope.go:122] "RemoveContainer" containerID="0a826660f48a81dce3fa56a603c55a8b4d793d9a7b338b4b2a4b110f73c0ca81"
Nov 02 14:45:22.693721 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:22.693667    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w_odh-ai-gateway-infra(b407ad8b-5856-4f5f-baf7-baebc7fe9233)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233"
Nov 02 14:45:22.694878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.694855    5034 generic.go:356] "Generic (PLEG): container finished" podID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerID="b31619433d72dedc9e15f8b41c914792829300d4b25a955c30014b498d45b6f4" exitCode=60
Nov 02 14:45:22.694990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.694945    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" event={"ID":"6d83cf56-9c22-47e5-aef4-087b779a2f65","Type":"ContainerDied","Data":"b31619433d72dedc9e15f8b41c914792829300d4b25a955c30014b498d45b6f4"}
Nov 02 14:45:22.695887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.695381    5034 scope.go:122] "RemoveContainer" containerID="b31619433d72dedc9e15f8b41c914792829300d4b25a955c30014b498d45b6f4"
Nov 02 14:45:22.695887 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:22.695760    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr_odh-ai-gateway-infra(6d83cf56-9c22-47e5-aef4-087b779a2f65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65"
Nov 02 14:45:22.722037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.718518    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" event={"ID":"5527c983-4d3e-4d15-8118-3f4f656b8051","Type":"ContainerStarted","Data":"06a2d70315619693ae2551f2703a696cf60a41d2ea02db34c48cedb04a33c89a"}
Nov 02 14:45:22.868217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.868148    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"]
Nov 02 14:45:22.930236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.927953    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"]
Nov 02 14:45:22.960918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.960884    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"]
Nov 02 14:45:22.999011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:22.996799    5034 scope.go:122] "RemoveContainer" containerID="3da29a98819607edd645993da30b78da4d5839d1f56af227c9e88d35e38c877c"
Nov 02 14:45:23.053170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.053141    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"]
Nov 02 14:45:23.108928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.108895    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"]
Nov 02 14:45:23.237227 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.237189    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:23.237826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.237469    5034 scope.go:122] "RemoveContainer" containerID="476ec33a22a342d6905570e705e2010e8b9d54a3a1db9203fe8f4543c1884edb"
Nov 02 14:45:23.265447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.265425    5034 scope.go:122] "RemoveContainer" containerID="ad2bdef45e40871ec0f623c6390af9c9175e021658ccdf4995ca09a0f8f852f3"
Nov 02 14:45:23.306014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.305979    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708b38e0-5f4a-4669-84e7-f74b02a3425e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708b38e0-5f4a-4669-84e7-f74b02a3425e-maas-api-service-ca\") pod \"708b38e0-5f4a-4669-84e7-f74b02a3425e\" (UID: \"708b38e0-5f4a-4669-84e7-f74b02a3425e\") "
Nov 02 14:45:23.307004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.306968    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708b38e0-5f4a-4669-84e7-f74b02a3425e-kube-api-access-bnkcn\" (UniqueName: \"kubernetes.io/projected/708b38e0-5f4a-4669-84e7-f74b02a3425e-kube-api-access-bnkcn\") pod \"708b38e0-5f4a-4669-84e7-f74b02a3425e\" (UID: \"708b38e0-5f4a-4669-84e7-f74b02a3425e\") "
Nov 02 14:45:23.324969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.323229    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708b38e0-5f4a-4669-84e7-f74b02a3425e-maas-api-service-ca" pod "708b38e0-5f4a-4669-84e7-f74b02a3425e" (UID: "708b38e0-5f4a-4669-84e7-f74b02a3425e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:23.324969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.324389    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/708b38e0-5f4a-4669-84e7-f74b02a3425e-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.325220 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.325091    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708b38e0-5f4a-4669-84e7-f74b02a3425e-kube-api-access-bnkcn" pod "708b38e0-5f4a-4669-84e7-f74b02a3425e" (UID: "708b38e0-5f4a-4669-84e7-f74b02a3425e"). InnerVolumeSpecName "kube-api-access-bnkcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:23.376247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.376007    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:23.378332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.378262    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:23.381665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.381568    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:23.388238 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.387485    5034 scope.go:122] "RemoveContainer" containerID="df33e2b42bd38d830f53eda889ecc2d5587e23c2adcfc95035c7dda85eb97ac9"
Nov 02 14:45:23.389349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.389330    5034 scope.go:122] "RemoveContainer" containerID="d0d0c1abafeb962fb562ca120517b253528fd967d06fd94889b79e38b8afb8f8"
Nov 02 14:45:23.411821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.408438    5034 scope.go:122] "RemoveContainer" containerID="56b01bc170f1d69346fc26d3e5b670d6a80db4c6538aa228c1f28878303334ed"
Nov 02 14:45:23.426485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.426449    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac656679-7557-495a-a88e-8b166be2da0e-kube-api-access-glhlx\" (UniqueName: \"kubernetes.io/projected/ac656679-7557-495a-a88e-8b166be2da0e-kube-api-access-glhlx\") pod \"ac656679-7557-495a-a88e-8b166be2da0e\" (UID: \"ac656679-7557-495a-a88e-8b166be2da0e\") "
Nov 02 14:45:23.426618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.426588    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac656679-7557-495a-a88e-8b166be2da0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac656679-7557-495a-a88e-8b166be2da0e-maas-api-service-ca\") pod \"ac656679-7557-495a-a88e-8b166be2da0e\" (UID: \"ac656679-7557-495a-a88e-8b166be2da0e\") "
Nov 02 14:45:23.426996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.426717    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-kube-api-access-m5v4q\" (UniqueName: \"kubernetes.io/projected/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-kube-api-access-m5v4q\") pod \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\" (UID: \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\") "
Nov 02 14:45:23.426996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.426884    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-maas-api-service-ca\") pod \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\" (UID: \"5b3eed1b-a630-4ee7-b3fc-de9d967c544d\") "
Nov 02 14:45:23.426996 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.426964    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b296f5f-1030-4705-998d-4459d0706cb0-kube-api-access-nrkdk\" (UniqueName: \"kubernetes.io/projected/7b296f5f-1030-4705-998d-4459d0706cb0-kube-api-access-nrkdk\") pod \"7b296f5f-1030-4705-998d-4459d0706cb0\" (UID: \"7b296f5f-1030-4705-998d-4459d0706cb0\") "
Nov 02 14:45:23.427304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.427080    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b296f5f-1030-4705-998d-4459d0706cb0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b296f5f-1030-4705-998d-4459d0706cb0-maas-api-service-ca\") pod \"7b296f5f-1030-4705-998d-4459d0706cb0\" (UID: \"7b296f5f-1030-4705-998d-4459d0706cb0\") "
Nov 02 14:45:23.428034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.427844    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-maas-api-service-ca" pod "5b3eed1b-a630-4ee7-b3fc-de9d967c544d" (UID: "5b3eed1b-a630-4ee7-b3fc-de9d967c544d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:23.428372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.428225    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac656679-7557-495a-a88e-8b166be2da0e-maas-api-service-ca" pod "ac656679-7557-495a-a88e-8b166be2da0e" (UID: "ac656679-7557-495a-a88e-8b166be2da0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:23.437830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.430272    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnkcn\" (UniqueName: \"kubernetes.io/projected/708b38e0-5f4a-4669-84e7-f74b02a3425e-kube-api-access-bnkcn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.437830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.436995    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b296f5f-1030-4705-998d-4459d0706cb0-maas-api-service-ca" pod "7b296f5f-1030-4705-998d-4459d0706cb0" (UID: "7b296f5f-1030-4705-998d-4459d0706cb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:23.456828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.456709    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac656679-7557-495a-a88e-8b166be2da0e-kube-api-access-glhlx" pod "ac656679-7557-495a-a88e-8b166be2da0e" (UID: "ac656679-7557-495a-a88e-8b166be2da0e"). InnerVolumeSpecName "kube-api-access-glhlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:23.456828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.456797    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b296f5f-1030-4705-998d-4459d0706cb0-kube-api-access-nrkdk" pod "7b296f5f-1030-4705-998d-4459d0706cb0" (UID: "7b296f5f-1030-4705-998d-4459d0706cb0"). InnerVolumeSpecName "kube-api-access-nrkdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:23.457300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.457263    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-kube-api-access-m5v4q" pod "5b3eed1b-a630-4ee7-b3fc-de9d967c544d" (UID: "5b3eed1b-a630-4ee7-b3fc-de9d967c544d"). InnerVolumeSpecName "kube-api-access-m5v4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:23.492385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.492314    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:23.496099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.496072    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:23.533545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.533515    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-kube-api-access-twplp\" (UniqueName: \"kubernetes.io/projected/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-kube-api-access-twplp\") pod \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\" (UID: \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\") "
Nov 02 14:45:23.533771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.533749    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98bfda5e-d485-4a81-835b-4c369b3666e4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bfda5e-d485-4a81-835b-4c369b3666e4-maas-api-service-ca\") pod \"98bfda5e-d485-4a81-835b-4c369b3666e4\" (UID: \"98bfda5e-d485-4a81-835b-4c369b3666e4\") "
Nov 02 14:45:23.533912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.533885    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bfda5e-d485-4a81-835b-4c369b3666e4-kube-api-access-pgpqg\" (UniqueName: \"kubernetes.io/projected/98bfda5e-d485-4a81-835b-4c369b3666e4-kube-api-access-pgpqg\") pod \"98bfda5e-d485-4a81-835b-4c369b3666e4\" (UID: \"98bfda5e-d485-4a81-835b-4c369b3666e4\") "
Nov 02 14:45:23.533980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.533948    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-maas-api-service-ca\") pod \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\" (UID: \"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f\") "
Nov 02 14:45:23.535030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.534997    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac656679-7557-495a-a88e-8b166be2da0e-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.535157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.535032    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5v4q\" (UniqueName: \"kubernetes.io/projected/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-kube-api-access-m5v4q\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.535157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.535049    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b3eed1b-a630-4ee7-b3fc-de9d967c544d-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.535157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.535061    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrkdk\" (UniqueName: \"kubernetes.io/projected/7b296f5f-1030-4705-998d-4459d0706cb0-kube-api-access-nrkdk\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.535157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.535077    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b296f5f-1030-4705-998d-4459d0706cb0-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.535157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.535090    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glhlx\" (UniqueName: \"kubernetes.io/projected/ac656679-7557-495a-a88e-8b166be2da0e-kube-api-access-glhlx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.535157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.535129    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bfda5e-d485-4a81-835b-4c369b3666e4-maas-api-service-ca" pod "98bfda5e-d485-4a81-835b-4c369b3666e4" (UID: "98bfda5e-d485-4a81-835b-4c369b3666e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:23.535892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.535855    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-maas-api-service-ca" pod "9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" (UID: "9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:23.537865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.537840    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bfda5e-d485-4a81-835b-4c369b3666e4-kube-api-access-pgpqg" pod "98bfda5e-d485-4a81-835b-4c369b3666e4" (UID: "98bfda5e-d485-4a81-835b-4c369b3666e4"). InnerVolumeSpecName "kube-api-access-pgpqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:23.539255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.539235    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-kube-api-access-twplp" pod "9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" (UID: "9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f"). InnerVolumeSpecName "kube-api-access-twplp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:23.557349 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.557320    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:45:23.636878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.636770    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-istio-token\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-istio-token\") pod \"d4b62861-322e-48a0-938d-539acb06cbcf\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") "
Nov 02 14:45:23.637084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.636909    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-data\") pod \"d4b62861-322e-48a0-938d-539acb06cbcf\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") "
Nov 02 14:45:23.637084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.636963    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-certs\") pod \"d4b62861-322e-48a0-938d-539acb06cbcf\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") "
Nov 02 14:45:23.637084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.636991    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b62861-322e-48a0-938d-539acb06cbcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b62861-322e-48a0-938d-539acb06cbcf-istiod-ca-cert\") pod \"d4b62861-322e-48a0-938d-539acb06cbcf\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") "
Nov 02 14:45:23.637084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637046    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-kube-api-access-nsrm6\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-kube-api-access-nsrm6\") pod \"d4b62861-322e-48a0-938d-539acb06cbcf\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") "
Nov 02 14:45:23.637334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637193    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-envoy\") pod \"d4b62861-322e-48a0-938d-539acb06cbcf\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") "
Nov 02 14:45:23.637334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637200    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-data" pod "d4b62861-322e-48a0-938d-539acb06cbcf" (UID: "d4b62861-322e-48a0-938d-539acb06cbcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:23.637334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637234    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-socket\") pod \"d4b62861-322e-48a0-938d-539acb06cbcf\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") "
Nov 02 14:45:23.637493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637368    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-certs" pod "d4b62861-322e-48a0-938d-539acb06cbcf" (UID: "d4b62861-322e-48a0-938d-539acb06cbcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:23.637493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637377    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-credential-socket\") pod \"d4b62861-322e-48a0-938d-539acb06cbcf\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") "
Nov 02 14:45:23.637590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637565    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-credential-socket" pod "d4b62861-322e-48a0-938d-539acb06cbcf" (UID: "d4b62861-322e-48a0-938d-539acb06cbcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:23.637652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637596    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4b62861-322e-48a0-938d-539acb06cbcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b62861-322e-48a0-938d-539acb06cbcf-istio-podinfo\") pod \"d4b62861-322e-48a0-938d-539acb06cbcf\" (UID: \"d4b62861-322e-48a0-938d-539acb06cbcf\") "
Nov 02 14:45:23.637803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637755    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-socket" pod "d4b62861-322e-48a0-938d-539acb06cbcf" (UID: "d4b62861-322e-48a0-938d-539acb06cbcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:23.637998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.637977    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b62861-322e-48a0-938d-539acb06cbcf-istiod-ca-cert" pod "d4b62861-322e-48a0-938d-539acb06cbcf" (UID: "d4b62861-322e-48a0-938d-539acb06cbcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:23.638487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.638456    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.638599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.638489    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bfda5e-d485-4a81-835b-4c369b3666e4-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.638599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.638507    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgpqg\" (UniqueName: \"kubernetes.io/projected/98bfda5e-d485-4a81-835b-4c369b3666e4-kube-api-access-pgpqg\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.638599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.638524    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.638599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.638538    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.638599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.638552    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.638599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.638565    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b62861-322e-48a0-938d-539acb06cbcf-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.638599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.638580    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.638599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.638595    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twplp\" (UniqueName: \"kubernetes.io/projected/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f-kube-api-access-twplp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.640187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.640156    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-istio-token" pod "d4b62861-322e-48a0-938d-539acb06cbcf" (UID: "d4b62861-322e-48a0-938d-539acb06cbcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:23.640304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.640287    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-envoy" pod "d4b62861-322e-48a0-938d-539acb06cbcf" (UID: "d4b62861-322e-48a0-938d-539acb06cbcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:23.640522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.640444    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-kube-api-access-nsrm6" pod "d4b62861-322e-48a0-938d-539acb06cbcf" (UID: "d4b62861-322e-48a0-938d-539acb06cbcf"). InnerVolumeSpecName "kube-api-access-nsrm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:23.640653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.640632    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4b62861-322e-48a0-938d-539acb06cbcf-istio-podinfo" pod "d4b62861-322e-48a0-938d-539acb06cbcf" (UID: "d4b62861-322e-48a0-938d-539acb06cbcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:45:23.736599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.736553    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" event={"ID":"86a9e4f4-79f9-4e98-be5e-f871f11b183d","Type":"ContainerStarted","Data":"99520363068cf3e7d011a27db8ff43f96785606a7a86de2b16406488839be460"}
Nov 02 14:45:23.738713 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.738687    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"
Nov 02 14:45:23.738919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.738724    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk" event={"ID":"9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f","Type":"ContainerDied","Data":"1ddb4b0a9a40024ce28d61fabfbed81e280bca73f1933d4df711200fd7169c17"}
Nov 02 14:45:23.738919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.738827    5034 scope.go:122] "RemoveContainer" containerID="1493a09ac19b138e2cc93dd3cb1ea8c5ab048f5cd21cadd254da1b9a43634676"
Nov 02 14:45:23.740033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.739998    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b62861-322e-48a0-938d-539acb06cbcf-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.740033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.740028    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.740248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.740041    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsrm6\" (UniqueName: \"kubernetes.io/projected/d4b62861-322e-48a0-938d-539acb06cbcf-kube-api-access-nsrm6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.740248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.740055    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b62861-322e-48a0-938d-539acb06cbcf-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:23.749833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.749802    5034 generic.go:356] "Generic (PLEG): container finished" podID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerID="ae43884f817d03f487ea36aa36e47bc4d911d1a439741a168eef0cabc9b9a3e5" exitCode=60
Nov 02 14:45:23.750129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.749898    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" event={"ID":"a29378f6-902d-4efb-977a-dd1f13dd5248","Type":"ContainerDied","Data":"ae43884f817d03f487ea36aa36e47bc4d911d1a439741a168eef0cabc9b9a3e5"}
Nov 02 14:45:23.751219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.750627    5034 scope.go:122] "RemoveContainer" containerID="ae43884f817d03f487ea36aa36e47bc4d911d1a439741a168eef0cabc9b9a3e5"
Nov 02 14:45:23.751219 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:23.751158    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn_odh-ai-gateway-infra(a29378f6-902d-4efb-977a-dd1f13dd5248)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248"
Nov 02 14:45:23.758587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.758548    5034 scope.go:122] "RemoveContainer" containerID="b81d119011cf51f51cc04c677154c8ccb5565f41c74ae32150f57561e060c27e"
Nov 02 14:45:23.762054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.760500    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt" event={"ID":"ac656679-7557-495a-a88e-8b166be2da0e","Type":"ContainerDied","Data":"3b67c93cafc6f5827b1c209749bf14d561176022e9ad87bd332ba972f02abdbd"}
Nov 02 14:45:23.762054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.760524    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"
Nov 02 14:45:23.767525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.766414    5034 generic.go:356] "Generic (PLEG): container finished" podID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerID="95d9a1abf319246cd7ba64038e30f48fe4cbd5f9a38153e81211e563bb9eda6f" exitCode=60
Nov 02 14:45:23.767525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.766517    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" event={"ID":"3238cdd3-2e70-456b-afe8-6cdbf2b20b89","Type":"ContainerDied","Data":"95d9a1abf319246cd7ba64038e30f48fe4cbd5f9a38153e81211e563bb9eda6f"}
Nov 02 14:45:23.767525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.767335    5034 scope.go:122] "RemoveContainer" containerID="95d9a1abf319246cd7ba64038e30f48fe4cbd5f9a38153e81211e563bb9eda6f"
Nov 02 14:45:23.767912 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:23.767881    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s_odh-ai-gateway-infra(3238cdd3-2e70-456b-afe8-6cdbf2b20b89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89"
Nov 02 14:45:23.785464 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.785429    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr" event={"ID":"7b296f5f-1030-4705-998d-4459d0706cb0","Type":"ContainerDied","Data":"4ea796295d39641e8cf7cc755635e47e6199fda3ee27924ef809bc1a66fb92a7"}
Nov 02 14:45:23.785561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.785544    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"
Nov 02 14:45:23.799798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.794810    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj" event={"ID":"5b3eed1b-a630-4ee7-b3fc-de9d967c544d","Type":"ContainerDied","Data":"6c5fe916a18697f285741f0cbcbea1a0acb34baff6639772009df91c10e0316d"}
Nov 02 14:45:23.799798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.794931    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"
Nov 02 14:45:23.799798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.798514    5034 generic.go:356] "Generic (PLEG): container finished" podID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerID="06a2d70315619693ae2551f2703a696cf60a41d2ea02db34c48cedb04a33c89a" exitCode=60
Nov 02 14:45:23.799798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.798603    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" event={"ID":"5527c983-4d3e-4d15-8118-3f4f656b8051","Type":"ContainerDied","Data":"06a2d70315619693ae2551f2703a696cf60a41d2ea02db34c48cedb04a33c89a"}
Nov 02 14:45:23.799798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.799150    5034 scope.go:122] "RemoveContainer" containerID="06a2d70315619693ae2551f2703a696cf60a41d2ea02db34c48cedb04a33c89a"
Nov 02 14:45:23.799798 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:23.799599    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb_odh-ai-gateway-infra(5527c983-4d3e-4d15-8118-3f4f656b8051)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051"
Nov 02 14:45:23.802581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.802556    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"
Nov 02 14:45:23.804173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.804135    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"]
Nov 02 14:45:23.805808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.804291    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq" event={"ID":"98bfda5e-d485-4a81-835b-4c369b3666e4","Type":"ContainerDied","Data":"ca84e7847d9c5480dc8541dbaf2b9313e081c2dd049c6a980f211ca94bb133d7"}
Nov 02 14:45:23.811711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.809966    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" event={"ID":"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7","Type":"ContainerStarted","Data":"9d895a2e01c6d38bc2abbb6888b48258ae9b2c918cca03deedf16f102ac12d1e"}
Nov 02 14:45:23.867933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.864476    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc" event={"ID":"708b38e0-5f4a-4669-84e7-f74b02a3425e","Type":"ContainerDied","Data":"5aec73989cce7f81a9e9f461e91a68e77e890ea7ce71e74c98e424eac38ec7db"}
Nov 02 14:45:23.867933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.864684    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"
Nov 02 14:45:23.938753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.938700    5034 generic.go:356] "Generic (PLEG): container finished" podID="d4b62861-322e-48a0-938d-539acb06cbcf" containerID="ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8" exitCode=0
Nov 02 14:45:23.938943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.938767    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" event={"ID":"d4b62861-322e-48a0-938d-539acb06cbcf","Type":"ContainerDied","Data":"ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8"}
Nov 02 14:45:23.938943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.938836    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x" event={"ID":"d4b62861-322e-48a0-938d-539acb06cbcf","Type":"ContainerDied","Data":"65d1c575608ee4d529916df7329af659366a4c102cdd3ca6d1b876911909f33d"}
Nov 02 14:45:23.939088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.938972    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"
Nov 02 14:45:24.008702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:23.987472    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"]
Nov 02 14:45:24.130685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.130519    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"]
Nov 02 14:45:24.151906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.151552    5034 scope.go:122] "RemoveContainer" containerID="d20919307627d31956761183b73e04f782333b46a9c323320b6c715479ff7e41"
Nov 02 14:45:24.154423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.154403    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"]
Nov 02 14:45:24.154647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.154625    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy" containerID="cri-o://3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574" gracePeriod=30
Nov 02 14:45:24.368658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.368623    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"]
Nov 02 14:45:24.406957 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.402699    5034 scope.go:122] "RemoveContainer" containerID="fa0819fba1215530776f58a8a21fbddc2e67a01804957d7c1c2650f59beffaf1"
Nov 02 14:45:24.406957 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.405020    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:24.406957 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.405188    5034 scope.go:122] "RemoveContainer" containerID="7a2ff66c32af1b2fa662a01cdc7d2131c447367fa7e8d6d8cad0203c7456a0a0"
Nov 02 14:45:24.529945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.529910    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dcb0f45-1ec6-43ac-a9af-571593753a63-kube-api-access-xgn6l\" (UniqueName: \"kubernetes.io/projected/4dcb0f45-1ec6-43ac-a9af-571593753a63-kube-api-access-xgn6l\") pod \"4dcb0f45-1ec6-43ac-a9af-571593753a63\" (UID: \"4dcb0f45-1ec6-43ac-a9af-571593753a63\") "
Nov 02 14:45:24.530215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.530187    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dcb0f45-1ec6-43ac-a9af-571593753a63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dcb0f45-1ec6-43ac-a9af-571593753a63-maas-api-service-ca\") pod \"4dcb0f45-1ec6-43ac-a9af-571593753a63\" (UID: \"4dcb0f45-1ec6-43ac-a9af-571593753a63\") "
Nov 02 14:45:24.534771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.534725    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dcb0f45-1ec6-43ac-a9af-571593753a63-maas-api-service-ca" pod "4dcb0f45-1ec6-43ac-a9af-571593753a63" (UID: "4dcb0f45-1ec6-43ac-a9af-571593753a63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:24.535325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.535288    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dcb0f45-1ec6-43ac-a9af-571593753a63-kube-api-access-xgn6l" pod "4dcb0f45-1ec6-43ac-a9af-571593753a63" (UID: "4dcb0f45-1ec6-43ac-a9af-571593753a63"). InnerVolumeSpecName "kube-api-access-xgn6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:24.543020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.542994    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"]
Nov 02 14:45:24.549770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.549266    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-c8f647-29893845-d7kfk"]
Nov 02 14:45:24.551276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.551209    5034 scope.go:122] "RemoveContainer" containerID="68d291693cf03d37186bd90f9e260ff6b3c01646bca11080ea0ff6408ac8b73f"
Nov 02 14:45:24.555323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.555306    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:24.575657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.575633    5034 scope.go:122] "RemoveContainer" containerID="5af0d06cade01907d37c219fecacdb2f92f9e2f3bd8f11aff7bedd221163c8cc"
Nov 02 14:45:24.632446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.630943    5034 scope.go:122] "RemoveContainer" containerID="8d47d5cd66be15c1eb5f773696d0863e3de5cb5d0276f054ab20dc8a6dc71808"
Nov 02 14:45:24.632446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.632135    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d83cf56-9c22-47e5-aef4-087b779a2f65-kube-api-access-vxwgg\" (UniqueName: \"kubernetes.io/projected/6d83cf56-9c22-47e5-aef4-087b779a2f65-kube-api-access-vxwgg\") pod \"6d83cf56-9c22-47e5-aef4-087b779a2f65\" (UID: \"6d83cf56-9c22-47e5-aef4-087b779a2f65\") "
Nov 02 14:45:24.632446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.632326    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d83cf56-9c22-47e5-aef4-087b779a2f65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83cf56-9c22-47e5-aef4-087b779a2f65-maas-api-service-ca\") pod \"6d83cf56-9c22-47e5-aef4-087b779a2f65\" (UID: \"6d83cf56-9c22-47e5-aef4-087b779a2f65\") "
Nov 02 14:45:24.633544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.633076    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d83cf56-9c22-47e5-aef4-087b779a2f65-maas-api-service-ca" pod "6d83cf56-9c22-47e5-aef4-087b779a2f65" (UID: "6d83cf56-9c22-47e5-aef4-087b779a2f65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:24.633544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.633410    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d83cf56-9c22-47e5-aef4-087b779a2f65-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.633544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.633432    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgn6l\" (UniqueName: \"kubernetes.io/projected/4dcb0f45-1ec6-43ac-a9af-571593753a63-kube-api-access-xgn6l\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.633544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.633447    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4dcb0f45-1ec6-43ac-a9af-571593753a63-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.637044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.637013    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d83cf56-9c22-47e5-aef4-087b779a2f65-kube-api-access-vxwgg" pod "6d83cf56-9c22-47e5-aef4-087b779a2f65" (UID: "6d83cf56-9c22-47e5-aef4-087b779a2f65"). InnerVolumeSpecName "kube-api-access-vxwgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:24.666737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.666710    5034 scope.go:122] "RemoveContainer" containerID="aec0492ad3bb5189eb40a860592824cde9841281193d43f3471e0bf322b3a460"
Nov 02 14:45:24.667881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.667864    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:24.680579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.680554    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:24.684524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.684499    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:24.685449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.685425    5034 scope.go:122] "RemoveContainer" containerID="1a963406a97f5b897636e9289cfd6c4b8a62cb887e2dbb6fecc3f41db734a81b"
Nov 02 14:45:24.698683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.698658    5034 scope.go:122] "RemoveContainer" containerID="ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8"
Nov 02 14:45:24.714161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.714131    5034 scope.go:122] "RemoveContainer" containerID="ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8"
Nov 02 14:45:24.714576 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:24.714553    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8\": container with ID starting with ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8 not found: ID does not exist" containerID="ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8"
Nov 02 14:45:24.714671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.714592    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8"} err="failed to get container status \"ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8\": rpc error: code = NotFound desc = could not find container \"ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8\": container with ID starting with ae9d78764d2fb419d3f16639948976525fe6ce2a296c0caa89dcaeb3de14a6c8 not found: ID does not exist"
Nov 02 14:45:24.765268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.765222    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b407ad8b-5856-4f5f-baf7-baebc7fe9233-kube-api-access-5wjl4\" (UniqueName: \"kubernetes.io/projected/b407ad8b-5856-4f5f-baf7-baebc7fe9233-kube-api-access-5wjl4\") pod \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\" (UID: \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\") "
Nov 02 14:45:24.766048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.766022    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b407ad8b-5856-4f5f-baf7-baebc7fe9233-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b407ad8b-5856-4f5f-baf7-baebc7fe9233-maas-api-service-ca\") pod \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\" (UID: \"b407ad8b-5856-4f5f-baf7-baebc7fe9233\") "
Nov 02 14:45:24.778916 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.778825    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxwgg\" (UniqueName: \"kubernetes.io/projected/6d83cf56-9c22-47e5-aef4-087b779a2f65-kube-api-access-vxwgg\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.780301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.779466    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b407ad8b-5856-4f5f-baf7-baebc7fe9233-maas-api-service-ca" pod "b407ad8b-5856-4f5f-baf7-baebc7fe9233" (UID: "b407ad8b-5856-4f5f-baf7-baebc7fe9233"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:24.787324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.787246    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b407ad8b-5856-4f5f-baf7-baebc7fe9233-kube-api-access-5wjl4" pod "b407ad8b-5856-4f5f-baf7-baebc7fe9233" (UID: "b407ad8b-5856-4f5f-baf7-baebc7fe9233"). InnerVolumeSpecName "kube-api-access-5wjl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:24.812704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.812522    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"]
Nov 02 14:45:24.820178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.820144    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2cddf2-29893845-d52hc"]
Nov 02 14:45:24.881481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.881440    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de34363d-7a31-4d79-8471-9b4e69d181d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de34363d-7a31-4d79-8471-9b4e69d181d5-maas-api-service-ca\") pod \"de34363d-7a31-4d79-8471-9b4e69d181d5\" (UID: \"de34363d-7a31-4d79-8471-9b4e69d181d5\") "
Nov 02 14:45:24.881674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.881611    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e9989de-be08-4242-944b-9db03854c4ed-kube-api-access-hkpt2\" (UniqueName: \"kubernetes.io/projected/0e9989de-be08-4242-944b-9db03854c4ed-kube-api-access-hkpt2\") pod \"0e9989de-be08-4242-944b-9db03854c4ed\" (UID: \"0e9989de-be08-4242-944b-9db03854c4ed\") "
Nov 02 14:45:24.881674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.881661    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e9989de-be08-4242-944b-9db03854c4ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9989de-be08-4242-944b-9db03854c4ed-maas-api-service-ca\") pod \"0e9989de-be08-4242-944b-9db03854c4ed\" (UID: \"0e9989de-be08-4242-944b-9db03854c4ed\") "
Nov 02 14:45:24.881819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.881727    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de34363d-7a31-4d79-8471-9b4e69d181d5-kube-api-access-bvhb2\" (UniqueName: \"kubernetes.io/projected/de34363d-7a31-4d79-8471-9b4e69d181d5-kube-api-access-bvhb2\") pod \"de34363d-7a31-4d79-8471-9b4e69d181d5\" (UID: \"de34363d-7a31-4d79-8471-9b4e69d181d5\") "
Nov 02 14:45:24.882435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.882411    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b407ad8b-5856-4f5f-baf7-baebc7fe9233-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.882563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.882439    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wjl4\" (UniqueName: \"kubernetes.io/projected/b407ad8b-5856-4f5f-baf7-baebc7fe9233-kube-api-access-5wjl4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.882862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.882829    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de34363d-7a31-4d79-8471-9b4e69d181d5-maas-api-service-ca" pod "de34363d-7a31-4d79-8471-9b4e69d181d5" (UID: "de34363d-7a31-4d79-8471-9b4e69d181d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:24.883288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.883265    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9989de-be08-4242-944b-9db03854c4ed-maas-api-service-ca" pod "0e9989de-be08-4242-944b-9db03854c4ed" (UID: "0e9989de-be08-4242-944b-9db03854c4ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:24.960800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.956330    5034 generic.go:356] "Generic (PLEG): container finished" podID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerID="99520363068cf3e7d011a27db8ff43f96785606a7a86de2b16406488839be460" exitCode=60
Nov 02 14:45:24.960800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.956471    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" event={"ID":"86a9e4f4-79f9-4e98-be5e-f871f11b183d","Type":"ContainerDied","Data":"99520363068cf3e7d011a27db8ff43f96785606a7a86de2b16406488839be460"}
Nov 02 14:45:24.960800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.956526    5034 scope.go:122] "RemoveContainer" containerID="e72a369c122d8ede1688d22861b4d15a19d55c9f516f9e64848208ac0d3f7d5f"
Nov 02 14:45:24.960800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.957158    5034 scope.go:122] "RemoveContainer" containerID="99520363068cf3e7d011a27db8ff43f96785606a7a86de2b16406488839be460"
Nov 02 14:45:24.960800 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:24.957731    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893845-cgqjf_odh-ai-gateway-infra(86a9e4f4-79f9-4e98-be5e-f871f11b183d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d"
Nov 02 14:45:24.962564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.961965    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn" event={"ID":"4dcb0f45-1ec6-43ac-a9af-571593753a63","Type":"ContainerDied","Data":"1a42536aa134dd643634630dbefdc8982e79a4d830b1c2020ffb839cff758c10"}
Nov 02 14:45:24.962564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.962009    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"]
Nov 02 14:45:24.962564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.962057    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"
Nov 02 14:45:24.964753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.964731    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"
Nov 02 14:45:24.965076 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.964772    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26" event={"ID":"de34363d-7a31-4d79-8471-9b4e69d181d5","Type":"ContainerDied","Data":"265b91b3ffe0b1f10d48e2d420e511e41540ef1e5dcdd845148a254141c06445"}
Nov 02 14:45:24.970093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.970065    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9989de-be08-4242-944b-9db03854c4ed-kube-api-access-hkpt2" pod "0e9989de-be08-4242-944b-9db03854c4ed" (UID: "0e9989de-be08-4242-944b-9db03854c4ed"). InnerVolumeSpecName "kube-api-access-hkpt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:24.972157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.972098    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de34363d-7a31-4d79-8471-9b4e69d181d5-kube-api-access-bvhb2" pod "de34363d-7a31-4d79-8471-9b4e69d181d5" (UID: "de34363d-7a31-4d79-8471-9b4e69d181d5"). InnerVolumeSpecName "kube-api-access-bvhb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:24.972635 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.972606    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" event={"ID":"6518f1f5-1243-49f7-8bf2-b174142e754a","Type":"ContainerStarted","Data":"ebabe5293d9e0bc6dfad1f766446f12b83ba9412ba838dfb1e2fdc651e741939"}
Nov 02 14:45:24.986362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.985472    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkpt2\" (UniqueName: \"kubernetes.io/projected/0e9989de-be08-4242-944b-9db03854c4ed-kube-api-access-hkpt2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.986362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.985533    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9989de-be08-4242-944b-9db03854c4ed-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.986362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.985550    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvhb2\" (UniqueName: \"kubernetes.io/projected/de34363d-7a31-4d79-8471-9b4e69d181d5-kube-api-access-bvhb2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.986362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.985564    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de34363d-7a31-4d79-8471-9b4e69d181d5-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:24.994215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.994155    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ce0f55-29893845-dp4tr"]
Nov 02 14:45:24.994486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:24.994471    5034 scope.go:122] "RemoveContainer" containerID="1c0b6edaf53d29427fa76dae079c9072e5d622c4c56184651428dcd511a97ba0"
Nov 02 14:45:25.003284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.003199    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"]
Nov 02 14:45:25.012882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.012157    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w" event={"ID":"b407ad8b-5856-4f5f-baf7-baebc7fe9233","Type":"ContainerDied","Data":"6a7dd31e0ad27d1de7f27af435f124a46464cc286cc7b71b1e637ad4bd9371cb"}
Nov 02 14:45:25.012882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.012271    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"
Nov 02 14:45:25.021751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.021729    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"]
Nov 02 14:45:25.023768 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.023744    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr" event={"ID":"6d83cf56-9c22-47e5-aef4-087b779a2f65","Type":"ContainerDied","Data":"9a8e5b0bb2a149f5c4fc97cdbb81bab30505b4f98b6e8adac0180855e8ffc8fa"}
Nov 02 14:45:25.024196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.024142    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"
Nov 02 14:45:25.030260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.029951    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-cbde78-29893845-z86dt"]
Nov 02 14:45:25.043120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.040245    5034 generic.go:356] "Generic (PLEG): container finished" podID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerID="9d895a2e01c6d38bc2abbb6888b48258ae9b2c918cca03deedf16f102ac12d1e" exitCode=60
Nov 02 14:45:25.043120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.040338    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" event={"ID":"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7","Type":"ContainerDied","Data":"9d895a2e01c6d38bc2abbb6888b48258ae9b2c918cca03deedf16f102ac12d1e"}
Nov 02 14:45:25.046872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.046557    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" event={"ID":"5bdbb368-93f7-4f30-89ef-881ec3528d6c","Type":"ContainerStarted","Data":"2c8c779b5adfeea25562431d9ab4efd410e5975181d6416a909943b6afec7e96"}
Nov 02 14:45:25.047742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.047649    5034 scope.go:122] "RemoveContainer" containerID="2c8c779b5adfeea25562431d9ab4efd410e5975181d6416a909943b6afec7e96"
Nov 02 14:45:25.049211 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:25.049150    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn_odh-ai-gateway-infra(5bdbb368-93f7-4f30-89ef-881ec3528d6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c"
Nov 02 14:45:25.059374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.057991    5034 generic.go:356] "Generic (PLEG): container finished" podID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerID="131b4ce9fe326587b8629674658a530b4531292221f320b176c12689b44bbfca" exitCode=60
Nov 02 14:45:25.059374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.058095    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8" event={"ID":"54f4abb8-f88b-42df-83c4-0dbffb278927","Type":"ContainerDied","Data":"131b4ce9fe326587b8629674658a530b4531292221f320b176c12689b44bbfca"}
Nov 02 14:45:25.059374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.058604    5034 scope.go:122] "RemoveContainer" containerID="131b4ce9fe326587b8629674658a530b4531292221f320b176c12689b44bbfca"
Nov 02 14:45:25.060109 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:25.060046    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8_odh-ai-gateway-infra(54f4abb8-f88b-42df-83c4-0dbffb278927)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927"
Nov 02 14:45:25.061895 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.061837    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"]
Nov 02 14:45:25.063623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.063561    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv" event={"ID":"0e9989de-be08-4242-944b-9db03854c4ed","Type":"ContainerDied","Data":"9996c239df406912ae540b85c3369a6326e34e364e46f8893638d05bc9c5a0d2"}
Nov 02 14:45:25.063623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.063602    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"
Nov 02 14:45:25.115949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.092402    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-0d1ca7-29893845-7ccgq"]
Nov 02 14:45:25.141112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.139929    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"]
Nov 02 14:45:25.141112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.140106    5034 scope.go:122] "RemoveContainer" containerID="d2c29047bd2cae7637b4623fbc40025b49718a6426fae85ce1a5845f35a26916"
Nov 02 14:45:25.148599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.148015    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4b7759-29893845-xhcjj"]
Nov 02 14:45:25.226576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.225451    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"]
Nov 02 14:45:25.227744 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.227690    5034 scope.go:122] "RemoveContainer" containerID="0a826660f48a81dce3fa56a603c55a8b4d793d9a7b338b4b2a4b110f73c0ca81"
Nov 02 14:45:25.232427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.232407    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-09d1a5-openshift-default-69bdf58dff-7qr2x"]
Nov 02 14:45:25.246898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.246720    5034 scope.go:122] "RemoveContainer" containerID="b31619433d72dedc9e15f8b41c914792829300d4b25a955c30014b498d45b6f4"
Nov 02 14:45:25.293677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.293494    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:25.301538 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.301504    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29378f6-902d-4efb-977a-dd1f13dd5248-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29378f6-902d-4efb-977a-dd1f13dd5248-maas-api-service-ca\") pod \"a29378f6-902d-4efb-977a-dd1f13dd5248\" (UID: \"a29378f6-902d-4efb-977a-dd1f13dd5248\") "
Nov 02 14:45:25.301692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.301612    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29378f6-902d-4efb-977a-dd1f13dd5248-kube-api-access-fzcxk\" (UniqueName: \"kubernetes.io/projected/a29378f6-902d-4efb-977a-dd1f13dd5248-kube-api-access-fzcxk\") pod \"a29378f6-902d-4efb-977a-dd1f13dd5248\" (UID: \"a29378f6-902d-4efb-977a-dd1f13dd5248\") "
Nov 02 14:45:25.303431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.303402    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29378f6-902d-4efb-977a-dd1f13dd5248-maas-api-service-ca" pod "a29378f6-902d-4efb-977a-dd1f13dd5248" (UID: "a29378f6-902d-4efb-977a-dd1f13dd5248"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:25.304052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.304024    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29378f6-902d-4efb-977a-dd1f13dd5248-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:25.305367 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.305346    5034 scope.go:122] "RemoveContainer" containerID="3c1a727b903e55fef6636bc699c1d354af0e3903cf2a120b78015d6e049a41b6"
Nov 02 14:45:25.306675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.306645    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29378f6-902d-4efb-977a-dd1f13dd5248-kube-api-access-fzcxk" pod "a29378f6-902d-4efb-977a-dd1f13dd5248" (UID: "a29378f6-902d-4efb-977a-dd1f13dd5248"). InnerVolumeSpecName "kube-api-access-fzcxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:25.328626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.328554    5034 scope.go:122] "RemoveContainer" containerID="df33e2b42bd38d830f53eda889ecc2d5587e23c2adcfc95035c7dda85eb97ac9"
Nov 02 14:45:25.361423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.361400    5034 scope.go:122] "RemoveContainer" containerID="de9d916d6518974f358f058b63de280845ae8e6d62a63a08aaebb1e328034472"
Nov 02 14:45:25.411644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.411608    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzcxk\" (UniqueName: \"kubernetes.io/projected/a29378f6-902d-4efb-977a-dd1f13dd5248-kube-api-access-fzcxk\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:25.416738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.416705    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" path="/var/lib/kubelet/pods/5b3eed1b-a630-4ee7-b3fc-de9d967c544d/volumes"
Nov 02 14:45:25.417507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.417485    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" path="/var/lib/kubelet/pods/708b38e0-5f4a-4669-84e7-f74b02a3425e/volumes"
Nov 02 14:45:25.420931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.418117    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" path="/var/lib/kubelet/pods/7b296f5f-1030-4705-998d-4459d0706cb0/volumes"
Nov 02 14:45:25.420931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.419166    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" path="/var/lib/kubelet/pods/98bfda5e-d485-4a81-835b-4c369b3666e4/volumes"
Nov 02 14:45:25.420931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.419738    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" path="/var/lib/kubelet/pods/9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f/volumes"
Nov 02 14:45:25.420931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.420320    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac656679-7557-495a-a88e-8b166be2da0e" path="/var/lib/kubelet/pods/ac656679-7557-495a-a88e-8b166be2da0e/volumes"
Nov 02 14:45:25.421474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.421449    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" path="/var/lib/kubelet/pods/d4b62861-322e-48a0-938d-539acb06cbcf/volumes"
Nov 02 14:45:25.423363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.423281    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:25.444819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.443325    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"]
Nov 02 14:45:25.455322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.452544    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4c2811-29893845-r77zv"]
Nov 02 14:45:25.472953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.472922    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"]
Nov 02 14:45:25.487843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.487486    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:25.489862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.489838    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:25.505919 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.505857    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"]
Nov 02 14:45:25.510834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.510799    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-16c822-29893845-fmnzr"]
Nov 02 14:45:25.512221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.512200    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5527c983-4d3e-4d15-8118-3f4f656b8051-kube-api-access-n66lc\" (UniqueName: \"kubernetes.io/projected/5527c983-4d3e-4d15-8118-3f4f656b8051-kube-api-access-n66lc\") pod \"5527c983-4d3e-4d15-8118-3f4f656b8051\" (UID: \"5527c983-4d3e-4d15-8118-3f4f656b8051\") "
Nov 02 14:45:25.512767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.512745    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-kube-api-access-mmvv9\" (UniqueName: \"kubernetes.io/projected/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-kube-api-access-mmvv9\") pod \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\" (UID: \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\") "
Nov 02 14:45:25.512955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.512936    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-maas-api-service-ca\") pod \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\" (UID: \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\") "
Nov 02 14:45:25.513060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.513008    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5527c983-4d3e-4d15-8118-3f4f656b8051-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527c983-4d3e-4d15-8118-3f4f656b8051-maas-api-service-ca\") pod \"5527c983-4d3e-4d15-8118-3f4f656b8051\" (UID: \"5527c983-4d3e-4d15-8118-3f4f656b8051\") "
Nov 02 14:45:25.513125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.513070    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-kube-api-access-kwgsr\" (UniqueName: \"kubernetes.io/projected/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-kube-api-access-kwgsr\") pod \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\" (UID: \"3238cdd3-2e70-456b-afe8-6cdbf2b20b89\") "
Nov 02 14:45:25.513888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.513247    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-maas-api-service-ca\") pod \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\" (UID: \"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7\") "
Nov 02 14:45:25.513888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.513496    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-maas-api-service-ca" pod "3238cdd3-2e70-456b-afe8-6cdbf2b20b89" (UID: "3238cdd3-2e70-456b-afe8-6cdbf2b20b89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:25.513888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.513724    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5527c983-4d3e-4d15-8118-3f4f656b8051-maas-api-service-ca" pod "5527c983-4d3e-4d15-8118-3f4f656b8051" (UID: "5527c983-4d3e-4d15-8118-3f4f656b8051"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:25.514158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.514130    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:25.514219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.514166    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5527c983-4d3e-4d15-8118-3f4f656b8051-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:25.514219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.514133    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-maas-api-service-ca" pod "c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" (UID: "c7ee5b3f-068d-40b7-aaf1-461643ff7fe7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:25.515275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.515249    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-kube-api-access-mmvv9" pod "c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" (UID: "c7ee5b3f-068d-40b7-aaf1-461643ff7fe7"). InnerVolumeSpecName "kube-api-access-mmvv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:25.516311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.516288    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5527c983-4d3e-4d15-8118-3f4f656b8051-kube-api-access-n66lc" pod "5527c983-4d3e-4d15-8118-3f4f656b8051" (UID: "5527c983-4d3e-4d15-8118-3f4f656b8051"). InnerVolumeSpecName "kube-api-access-n66lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:25.517609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.517582    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-kube-api-access-kwgsr" pod "3238cdd3-2e70-456b-afe8-6cdbf2b20b89" (UID: "3238cdd3-2e70-456b-afe8-6cdbf2b20b89"). InnerVolumeSpecName "kube-api-access-kwgsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:25.616155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.616122    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwgsr\" (UniqueName: \"kubernetes.io/projected/3238cdd3-2e70-456b-afe8-6cdbf2b20b89-kube-api-access-kwgsr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:25.616155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.616153    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:25.616155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.616163    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n66lc\" (UniqueName: \"kubernetes.io/projected/5527c983-4d3e-4d15-8118-3f4f656b8051-kube-api-access-n66lc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:25.616448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.616172    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvv9\" (UniqueName: \"kubernetes.io/projected/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7-kube-api-access-mmvv9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:25.696750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.696706    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"]
Nov 02 14:45:25.704515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.704485    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2daae4-29893845-2bw6w"]
Nov 02 14:45:25.745857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.745770    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"]
Nov 02 14:45:25.750732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.750703    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-0969b1-29893845-nptkn"]
Nov 02 14:45:25.962897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.962809    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"]
Nov 02 14:45:25.977887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:25.977845    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ea3e8f-29893845-6wz26"]
Nov 02 14:45:26.083636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.083596    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb" event={"ID":"5527c983-4d3e-4d15-8118-3f4f656b8051","Type":"ContainerDied","Data":"6d5fe9442bdd8a972893ac7b9d21a7006d95f2d3baec5a9024665bf27aaf87bf"}
Nov 02 14:45:26.083874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.083686    5034 scope.go:122] "RemoveContainer" containerID="06a2d70315619693ae2551f2703a696cf60a41d2ea02db34c48cedb04a33c89a"
Nov 02 14:45:26.083874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.083852    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"
Nov 02 14:45:26.090373 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.090316    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2" event={"ID":"c7ee5b3f-068d-40b7-aaf1-461643ff7fe7","Type":"ContainerDied","Data":"15c845092151a593d472ff5b587c91382f869fd3108aa97b89c0ccff3af488df"}
Nov 02 14:45:26.091097 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.091078    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"
Nov 02 14:45:26.093133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.093108    5034 generic.go:356] "Generic (PLEG): container finished" podID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerID="2c8c779b5adfeea25562431d9ab4efd410e5975181d6416a909943b6afec7e96" exitCode=60
Nov 02 14:45:26.093267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.093202    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" event={"ID":"5bdbb368-93f7-4f30-89ef-881ec3528d6c","Type":"ContainerDied","Data":"2c8c779b5adfeea25562431d9ab4efd410e5975181d6416a909943b6afec7e96"}
Nov 02 14:45:26.093978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.093856    5034 scope.go:122] "RemoveContainer" containerID="2c8c779b5adfeea25562431d9ab4efd410e5975181d6416a909943b6afec7e96"
Nov 02 14:45:26.094356 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:26.094308    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn_odh-ai-gateway-infra(5bdbb368-93f7-4f30-89ef-881ec3528d6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c"
Nov 02 14:45:26.111096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.110295    5034 scope.go:122] "RemoveContainer" containerID="9d895a2e01c6d38bc2abbb6888b48258ae9b2c918cca03deedf16f102ac12d1e"
Nov 02 14:45:26.125936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.125908    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn" event={"ID":"a29378f6-902d-4efb-977a-dd1f13dd5248","Type":"ContainerDied","Data":"5f6a4e95b24e551eeb840b70c8ce46f0ac57a6f55b2ddcbc0261643894e5f952"}
Nov 02 14:45:26.133705 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.125919    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"
Nov 02 14:45:26.136631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.136598    5034 generic.go:356] "Generic (PLEG): container finished" podID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerID="4c49d53ebdacc6f081c551bf4607ce65f89f43a746b4d2e9c2c6de0292b45ee2" exitCode=60
Nov 02 14:45:26.136745 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.136697    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" event={"ID":"c63a0f7e-2b21-46b1-80d2-61d439daf047","Type":"ContainerDied","Data":"4c49d53ebdacc6f081c551bf4607ce65f89f43a746b4d2e9c2c6de0292b45ee2"}
Nov 02 14:45:26.137254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.137232    5034 scope.go:122] "RemoveContainer" containerID="4c49d53ebdacc6f081c551bf4607ce65f89f43a746b4d2e9c2c6de0292b45ee2"
Nov 02 14:45:26.137645 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:26.137619    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x_odh-ai-gateway-infra(c63a0f7e-2b21-46b1-80d2-61d439daf047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047"
Nov 02 14:45:26.142379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.142290    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" event={"ID":"6518f1f5-1243-49f7-8bf2-b174142e754a","Type":"ContainerDied","Data":"ebabe5293d9e0bc6dfad1f766446f12b83ba9412ba838dfb1e2fdc651e741939"}
Nov 02 14:45:26.142558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.142523    5034 generic.go:356] "Generic (PLEG): container finished" podID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerID="ebabe5293d9e0bc6dfad1f766446f12b83ba9412ba838dfb1e2fdc651e741939" exitCode=60
Nov 02 14:45:26.142842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.142769    5034 scope.go:122] "RemoveContainer" containerID="ebabe5293d9e0bc6dfad1f766446f12b83ba9412ba838dfb1e2fdc651e741939"
Nov 02 14:45:26.143236 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:26.143200    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m_odh-ai-gateway-infra(6518f1f5-1243-49f7-8bf2-b174142e754a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a"
Nov 02 14:45:26.144030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.144013    5034 scope.go:122] "RemoveContainer" containerID="d0d0c1abafeb962fb562ca120517b253528fd967d06fd94889b79e38b8afb8f8"
Nov 02 14:45:26.144771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.144748    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s" event={"ID":"3238cdd3-2e70-456b-afe8-6cdbf2b20b89","Type":"ContainerDied","Data":"c152d54b2d26326414e5c3fdc0184af9efd6de905a7805fd18f804382776c68a"}
Nov 02 14:45:26.144893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.144876    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"
Nov 02 14:45:26.172345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.172318    5034 scope.go:122] "RemoveContainer" containerID="ae43884f817d03f487ea36aa36e47bc4d911d1a439741a168eef0cabc9b9a3e5"
Nov 02 14:45:26.188890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.188801    5034 scope.go:122] "RemoveContainer" containerID="7a2ff66c32af1b2fa662a01cdc7d2131c447367fa7e8d6d8cad0203c7456a0a0"
Nov 02 14:45:26.208497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.208466    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"]
Nov 02 14:45:26.215376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.215340    5034 scope.go:122] "RemoveContainer" containerID="56b01bc170f1d69346fc26d3e5b670d6a80db4c6538aa228c1f28878303334ed"
Nov 02 14:45:26.218890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.218857    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-e1136e-29893845-vvdcn"]
Nov 02 14:45:26.235875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.235854    5034 scope.go:122] "RemoveContainer" containerID="95d9a1abf319246cd7ba64038e30f48fe4cbd5f9a38153e81211e563bb9eda6f"
Nov 02 14:45:26.291532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.291503    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"]
Nov 02 14:45:26.361803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.360875    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:26.383294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.383259    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"]
Nov 02 14:45:26.431583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.430667    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a9e4f4-79f9-4e98-be5e-f871f11b183d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a9e4f4-79f9-4e98-be5e-f871f11b183d-maas-api-service-ca\") pod \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\" (UID: \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\") "
Nov 02 14:45:26.431583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.430930    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a9e4f4-79f9-4e98-be5e-f871f11b183d-kube-api-access-mq9tx\" (UniqueName: \"kubernetes.io/projected/86a9e4f4-79f9-4e98-be5e-f871f11b183d-kube-api-access-mq9tx\") pod \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\" (UID: \"86a9e4f4-79f9-4e98-be5e-f871f11b183d\") "
Nov 02 14:45:26.431583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.431054    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a9e4f4-79f9-4e98-be5e-f871f11b183d-maas-api-service-ca" pod "86a9e4f4-79f9-4e98-be5e-f871f11b183d" (UID: "86a9e4f4-79f9-4e98-be5e-f871f11b183d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:26.431583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.431549    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86a9e4f4-79f9-4e98-be5e-f871f11b183d-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:26.434487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.434447    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a9e4f4-79f9-4e98-be5e-f871f11b183d-kube-api-access-mq9tx" pod "86a9e4f4-79f9-4e98-be5e-f871f11b183d" (UID: "86a9e4f4-79f9-4e98-be5e-f871f11b183d"). InnerVolumeSpecName "kube-api-access-mq9tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:26.439923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.439892    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"]
Nov 02 14:45:26.446101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.446075    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a4809a-29893845-4zwwb"]
Nov 02 14:45:26.486536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.483850    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"]
Nov 02 14:45:26.492523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.492490    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-fcefcc-29893845-scxh2"]
Nov 02 14:45:26.534108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.534072    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq9tx\" (UniqueName: \"kubernetes.io/projected/86a9e4f4-79f9-4e98-be5e-f871f11b183d-kube-api-access-mq9tx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:26.552853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.552820    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"]
Nov 02 14:45:26.578911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.578529    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"]
Nov 02 14:45:26.586771 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.586735    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:26.593887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.593859    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2d13f4-29893845-bd85s"]
Nov 02 14:45:26.647675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.646442    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f4abb8-f88b-42df-83c4-0dbffb278927-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4abb8-f88b-42df-83c4-0dbffb278927-maas-api-service-ca\") pod \"54f4abb8-f88b-42df-83c4-0dbffb278927\" (UID: \"54f4abb8-f88b-42df-83c4-0dbffb278927\") "
Nov 02 14:45:26.647675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.646607    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f4abb8-f88b-42df-83c4-0dbffb278927-kube-api-access-bdvfb\" (UniqueName: \"kubernetes.io/projected/54f4abb8-f88b-42df-83c4-0dbffb278927-kube-api-access-bdvfb\") pod \"54f4abb8-f88b-42df-83c4-0dbffb278927\" (UID: \"54f4abb8-f88b-42df-83c4-0dbffb278927\") "
Nov 02 14:45:26.650927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.648152    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f4abb8-f88b-42df-83c4-0dbffb278927-maas-api-service-ca" pod "54f4abb8-f88b-42df-83c4-0dbffb278927" (UID: "54f4abb8-f88b-42df-83c4-0dbffb278927"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:26.657753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.657699    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f4abb8-f88b-42df-83c4-0dbffb278927-kube-api-access-bdvfb" pod "54f4abb8-f88b-42df-83c4-0dbffb278927" (UID: "54f4abb8-f88b-42df-83c4-0dbffb278927"). InnerVolumeSpecName "kube-api-access-bdvfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:26.749345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.749265    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54f4abb8-f88b-42df-83c4-0dbffb278927-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:26.749345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:26.749292    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdvfb\" (UniqueName: \"kubernetes.io/projected/54f4abb8-f88b-42df-83c4-0dbffb278927-kube-api-access-bdvfb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:27.159547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.159474    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"
Nov 02 14:45:27.159547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.159507    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8" event={"ID":"54f4abb8-f88b-42df-83c4-0dbffb278927","Type":"ContainerDied","Data":"96066f78af50b3a20e8b08d18259f3379aa7ac91f5e6de0874d30eb3e837cc90"}
Nov 02 14:45:27.160172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.159570    5034 scope.go:122] "RemoveContainer" containerID="131b4ce9fe326587b8629674658a530b4531292221f320b176c12689b44bbfca"
Nov 02 14:45:27.161236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.161212    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"
Nov 02 14:45:27.161343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.161232    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf" event={"ID":"86a9e4f4-79f9-4e98-be5e-f871f11b183d","Type":"ContainerDied","Data":"7877fb7719b32c468ec4ba13ecb523e52ce3375551d660de8e20d250da822607"}
Nov 02 14:45:27.211745 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.211725    5034 scope.go:122] "RemoveContainer" containerID="99520363068cf3e7d011a27db8ff43f96785606a7a86de2b16406488839be460"
Nov 02 14:45:27.262462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.262426    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"]
Nov 02 14:45:27.269810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.269645    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d39fa6-29893845-96ls8"]
Nov 02 14:45:27.305112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.305085    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"]
Nov 02 14:45:27.433860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.430415    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e9989de-be08-4242-944b-9db03854c4ed" path="/var/lib/kubelet/pods/0e9989de-be08-4242-944b-9db03854c4ed/volumes"
Nov 02 14:45:27.433860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.431191    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" path="/var/lib/kubelet/pods/3238cdd3-2e70-456b-afe8-6cdbf2b20b89/volumes"
Nov 02 14:45:27.433860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.431979    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" path="/var/lib/kubelet/pods/4dcb0f45-1ec6-43ac-a9af-571593753a63/volumes"
Nov 02 14:45:27.434149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.433896    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" path="/var/lib/kubelet/pods/54f4abb8-f88b-42df-83c4-0dbffb278927/volumes"
Nov 02 14:45:27.435509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.434483    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" path="/var/lib/kubelet/pods/5527c983-4d3e-4d15-8118-3f4f656b8051/volumes"
Nov 02 14:45:27.435509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.435134    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" path="/var/lib/kubelet/pods/6d83cf56-9c22-47e5-aef4-087b779a2f65/volumes"
Nov 02 14:45:27.437268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.436213    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" path="/var/lib/kubelet/pods/a29378f6-902d-4efb-977a-dd1f13dd5248/volumes"
Nov 02 14:45:27.437268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.436832    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" path="/var/lib/kubelet/pods/b407ad8b-5856-4f5f-baf7-baebc7fe9233/volumes"
Nov 02 14:45:27.437847 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.437493    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" path="/var/lib/kubelet/pods/c7ee5b3f-068d-40b7-aaf1-461643ff7fe7/volumes"
Nov 02 14:45:27.439012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.438457    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" path="/var/lib/kubelet/pods/de34363d-7a31-4d79-8471-9b4e69d181d5/volumes"
Nov 02 14:45:27.439113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.439012    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"]
Nov 02 14:45:27.439113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.439058    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893845-cgqjf"]
Nov 02 14:45:27.754318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.748625    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:27.774656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.773632    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:27.789133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.789109    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:27.838658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.838629    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6518f1f5-1243-49f7-8bf2-b174142e754a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6518f1f5-1243-49f7-8bf2-b174142e754a-maas-api-service-ca\") pod \"6518f1f5-1243-49f7-8bf2-b174142e754a\" (UID: \"6518f1f5-1243-49f7-8bf2-b174142e754a\") "
Nov 02 14:45:27.838867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.838730    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63a0f7e-2b21-46b1-80d2-61d439daf047-kube-api-access-v9smp\" (UniqueName: \"kubernetes.io/projected/c63a0f7e-2b21-46b1-80d2-61d439daf047-kube-api-access-v9smp\") pod \"c63a0f7e-2b21-46b1-80d2-61d439daf047\" (UID: \"c63a0f7e-2b21-46b1-80d2-61d439daf047\") "
Nov 02 14:45:27.838867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.838834    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63a0f7e-2b21-46b1-80d2-61d439daf047-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63a0f7e-2b21-46b1-80d2-61d439daf047-maas-api-service-ca\") pod \"c63a0f7e-2b21-46b1-80d2-61d439daf047\" (UID: \"c63a0f7e-2b21-46b1-80d2-61d439daf047\") "
Nov 02 14:45:27.838990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.838869    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6518f1f5-1243-49f7-8bf2-b174142e754a-kube-api-access-9vc7z\" (UniqueName: \"kubernetes.io/projected/6518f1f5-1243-49f7-8bf2-b174142e754a-kube-api-access-9vc7z\") pod \"6518f1f5-1243-49f7-8bf2-b174142e754a\" (UID: \"6518f1f5-1243-49f7-8bf2-b174142e754a\") "
Nov 02 14:45:27.838990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.838966    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bdbb368-93f7-4f30-89ef-881ec3528d6c-kube-api-access-tx7sh\" (UniqueName: \"kubernetes.io/projected/5bdbb368-93f7-4f30-89ef-881ec3528d6c-kube-api-access-tx7sh\") pod \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\" (UID: \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\") "
Nov 02 14:45:27.839106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.839048    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bdbb368-93f7-4f30-89ef-881ec3528d6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdbb368-93f7-4f30-89ef-881ec3528d6c-maas-api-service-ca\") pod \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\" (UID: \"5bdbb368-93f7-4f30-89ef-881ec3528d6c\") "
Nov 02 14:45:27.839550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.839191    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6518f1f5-1243-49f7-8bf2-b174142e754a-maas-api-service-ca" pod "6518f1f5-1243-49f7-8bf2-b174142e754a" (UID: "6518f1f5-1243-49f7-8bf2-b174142e754a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:27.839669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.839544    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdbb368-93f7-4f30-89ef-881ec3528d6c-maas-api-service-ca" pod "5bdbb368-93f7-4f30-89ef-881ec3528d6c" (UID: "5bdbb368-93f7-4f30-89ef-881ec3528d6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:27.840228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.840203    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6518f1f5-1243-49f7-8bf2-b174142e754a-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:27.840302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.840232    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdbb368-93f7-4f30-89ef-881ec3528d6c-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:27.840369 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.840354    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63a0f7e-2b21-46b1-80d2-61d439daf047-maas-api-service-ca" pod "c63a0f7e-2b21-46b1-80d2-61d439daf047" (UID: "c63a0f7e-2b21-46b1-80d2-61d439daf047"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:27.854457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.854416    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdbb368-93f7-4f30-89ef-881ec3528d6c-kube-api-access-tx7sh" pod "5bdbb368-93f7-4f30-89ef-881ec3528d6c" (UID: "5bdbb368-93f7-4f30-89ef-881ec3528d6c"). InnerVolumeSpecName "kube-api-access-tx7sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:27.856858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.856828    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63a0f7e-2b21-46b1-80d2-61d439daf047-kube-api-access-v9smp" pod "c63a0f7e-2b21-46b1-80d2-61d439daf047" (UID: "c63a0f7e-2b21-46b1-80d2-61d439daf047"). InnerVolumeSpecName "kube-api-access-v9smp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:27.858343 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.858313    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6518f1f5-1243-49f7-8bf2-b174142e754a-kube-api-access-9vc7z" pod "6518f1f5-1243-49f7-8bf2-b174142e754a" (UID: "6518f1f5-1243-49f7-8bf2-b174142e754a"). InnerVolumeSpecName "kube-api-access-9vc7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:27.942526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.942489    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9smp\" (UniqueName: \"kubernetes.io/projected/c63a0f7e-2b21-46b1-80d2-61d439daf047-kube-api-access-v9smp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:27.942526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.942527    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c63a0f7e-2b21-46b1-80d2-61d439daf047-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:27.942736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.942544    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vc7z\" (UniqueName: \"kubernetes.io/projected/6518f1f5-1243-49f7-8bf2-b174142e754a-kube-api-access-9vc7z\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:27.942736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:27.942562    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx7sh\" (UniqueName: \"kubernetes.io/projected/5bdbb368-93f7-4f30-89ef-881ec3528d6c-kube-api-access-tx7sh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:28.242900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.242871    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"
Nov 02 14:45:28.243457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.242868    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn" event={"ID":"5bdbb368-93f7-4f30-89ef-881ec3528d6c","Type":"ContainerDied","Data":"c650fa7ca021bc12907bdcaa54164771d98f1a2659585afe57eb85d3907c2171"}
Nov 02 14:45:28.243457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.243033    5034 scope.go:122] "RemoveContainer" containerID="2c8c779b5adfeea25562431d9ab4efd410e5975181d6416a909943b6afec7e96"
Nov 02 14:45:28.247089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.247065    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"
Nov 02 14:45:28.247205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.247078    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x" event={"ID":"c63a0f7e-2b21-46b1-80d2-61d439daf047","Type":"ContainerDied","Data":"918915facc6d6597cf4c19092d39650a8febbfe9f2b434b4b6d13bae66cb0f5e"}
Nov 02 14:45:28.248670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.248644    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"
Nov 02 14:45:28.248790 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.248676    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m" event={"ID":"6518f1f5-1243-49f7-8bf2-b174142e754a","Type":"ContainerDied","Data":"28a6c166ce5a3049e1ea50f9a33acd4f677ab2a401b8616be4a6939a317ef8fc"}
Nov 02 14:45:28.259589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.259570    5034 scope.go:122] "RemoveContainer" containerID="4c49d53ebdacc6f081c551bf4607ce65f89f43a746b4d2e9c2c6de0292b45ee2"
Nov 02 14:45:28.270748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.270719    5034 scope.go:122] "RemoveContainer" containerID="ebabe5293d9e0bc6dfad1f766446f12b83ba9412ba838dfb1e2fdc651e741939"
Nov 02 14:45:28.325058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.325014    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"]
Nov 02 14:45:28.330161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.330125    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bc34db-29893845-k24mn"]
Nov 02 14:45:28.342977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.342947    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"]
Nov 02 14:45:28.348546 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.348519    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-21baa7-29893845-ckk9x"]
Nov 02 14:45:28.424218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.424182    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"]
Nov 02 14:45:28.431364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:28.431334    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-80133f-29893845-5pz6m"]
Nov 02 14:45:29.406479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.405497    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" path="/var/lib/kubelet/pods/5bdbb368-93f7-4f30-89ef-881ec3528d6c/volumes"
Nov 02 14:45:29.406479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.406362    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" path="/var/lib/kubelet/pods/6518f1f5-1243-49f7-8bf2-b174142e754a/volumes"
Nov 02 14:45:29.407407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.407127    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" path="/var/lib/kubelet/pods/86a9e4f4-79f9-4e98-be5e-f871f11b183d/volumes"
Nov 02 14:45:29.408438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.408158    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" path="/var/lib/kubelet/pods/c63a0f7e-2b21-46b1-80d2-61d439daf047/volumes"
Nov 02 14:45:29.484088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.484056    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:45:29.568977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.568938    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-credential-socket\") pod \"65501c91-8e19-4ee7-bb4c-ec746643d814\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") "
Nov 02 14:45:29.569180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.569017    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-kube-api-access-d79x8\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-kube-api-access-d79x8\") pod \"65501c91-8e19-4ee7-bb4c-ec746643d814\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") "
Nov 02 14:45:29.569180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.569084    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-token\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-token\") pod \"65501c91-8e19-4ee7-bb4c-ec746643d814\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") "
Nov 02 14:45:29.569334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.569204    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-envoy\") pod \"65501c91-8e19-4ee7-bb4c-ec746643d814\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") "
Nov 02 14:45:29.569334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.569247    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-credential-socket" pod "65501c91-8e19-4ee7-bb4c-ec746643d814" (UID: "65501c91-8e19-4ee7-bb4c-ec746643d814"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:29.569456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.569361    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-podinfo\") pod \"65501c91-8e19-4ee7-bb4c-ec746643d814\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") "
Nov 02 14:45:29.569456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.569393    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65501c91-8e19-4ee7-bb4c-ec746643d814-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65501c91-8e19-4ee7-bb4c-ec746643d814-istiod-ca-cert\") pod \"65501c91-8e19-4ee7-bb4c-ec746643d814\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") "
Nov 02 14:45:29.569565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.569527    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-data\") pod \"65501c91-8e19-4ee7-bb4c-ec746643d814\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") "
Nov 02 14:45:29.569624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.569572    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-certs\") pod \"65501c91-8e19-4ee7-bb4c-ec746643d814\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") "
Nov 02 14:45:29.569624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.569603    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-socket\") pod \"65501c91-8e19-4ee7-bb4c-ec746643d814\" (UID: \"65501c91-8e19-4ee7-bb4c-ec746643d814\") "
Nov 02 14:45:29.570346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.570138    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-data" pod "65501c91-8e19-4ee7-bb4c-ec746643d814" (UID: "65501c91-8e19-4ee7-bb4c-ec746643d814"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:29.570824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.570596    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:29.570824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.570628    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:29.571412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.571129    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65501c91-8e19-4ee7-bb4c-ec746643d814-istiod-ca-cert" pod "65501c91-8e19-4ee7-bb4c-ec746643d814" (UID: "65501c91-8e19-4ee7-bb4c-ec746643d814"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:29.575604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.575576    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-kube-api-access-d79x8" pod "65501c91-8e19-4ee7-bb4c-ec746643d814" (UID: "65501c91-8e19-4ee7-bb4c-ec746643d814"). InnerVolumeSpecName "kube-api-access-d79x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:29.575837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.575795    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-podinfo" pod "65501c91-8e19-4ee7-bb4c-ec746643d814" (UID: "65501c91-8e19-4ee7-bb4c-ec746643d814"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:45:29.576866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.576830    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-envoy" pod "65501c91-8e19-4ee7-bb4c-ec746643d814" (UID: "65501c91-8e19-4ee7-bb4c-ec746643d814"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:29.578481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.578454    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-token" pod "65501c91-8e19-4ee7-bb4c-ec746643d814" (UID: "65501c91-8e19-4ee7-bb4c-ec746643d814"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:29.579044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.579008    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-certs" pod "65501c91-8e19-4ee7-bb4c-ec746643d814" (UID: "65501c91-8e19-4ee7-bb4c-ec746643d814"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:29.580031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.580007    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-socket" pod "65501c91-8e19-4ee7-bb4c-ec746643d814" (UID: "65501c91-8e19-4ee7-bb4c-ec746643d814"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:45:29.672086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.672048    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:29.672086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.672084    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65501c91-8e19-4ee7-bb4c-ec746643d814-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:29.672310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.672099    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:29.672310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.672111    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:29.672310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.672125    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79x8\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-kube-api-access-d79x8\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:29.672310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.672140    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:29.672310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:29.672152    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65501c91-8e19-4ee7-bb4c-ec746643d814-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:30.274943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.274908    5034 generic.go:356] "Generic (PLEG): container finished" podID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerID="3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574" exitCode=0
Nov 02 14:45:30.275136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.274956    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"
Nov 02 14:45:30.275136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.274999    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" event={"ID":"65501c91-8e19-4ee7-bb4c-ec746643d814","Type":"ContainerDied","Data":"3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574"}
Nov 02 14:45:30.275136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.275064    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp" event={"ID":"65501c91-8e19-4ee7-bb4c-ec746643d814","Type":"ContainerDied","Data":"6ef555936a1f15048ce4a8262f8b766282d7bf577c347670339adc9ecaa1d75a"}
Nov 02 14:45:30.275136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.275093    5034 scope.go:122] "RemoveContainer" containerID="3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574"
Nov 02 14:45:30.291390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.291222    5034 scope.go:122] "RemoveContainer" containerID="1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e"
Nov 02 14:45:30.305830 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.305808    5034 scope.go:122] "RemoveContainer" containerID="3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574"
Nov 02 14:45:30.306167 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:30.306146    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574\": container with ID starting with 3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574 not found: ID does not exist" containerID="3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574"
Nov 02 14:45:30.306231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.306174    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574"} err="failed to get container status \"3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574\": rpc error: code = NotFound desc = could not find container \"3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574\": container with ID starting with 3ed0d2379776348412d6e98865ee810c05850b83cf53126a88319cf19d571574 not found: ID does not exist"
Nov 02 14:45:30.306231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.306191    5034 scope.go:122] "RemoveContainer" containerID="1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e"
Nov 02 14:45:30.306466 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:30.306446    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e\": container with ID starting with 1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e not found: ID does not exist" containerID="1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e"
Nov 02 14:45:30.306517 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.306470    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e"} err="failed to get container status \"1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e\": rpc error: code = NotFound desc = could not find container \"1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e\": container with ID starting with 1dd1c47aa6641cf52d3b62a632cad6864b82b583f8b1d98acf3002536103907e not found: ID does not exist"
Nov 02 14:45:30.349224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.349194    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"]
Nov 02 14:45:30.362036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:30.362004    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e1fcb4-openshift-default-57cf4bd49-j7lsp"]
Nov 02 14:45:31.192238 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:31.192202    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"]
Nov 02 14:45:31.399741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:31.399701    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" path="/var/lib/kubelet/pods/65501c91-8e19-4ee7-bb4c-ec746643d814/volumes"
Nov 02 14:45:34.652493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.652460    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"]
Nov 02 14:45:34.653307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653289    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerName="cleanup"
Nov 02 14:45:34.653361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653308    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerName="cleanup"
Nov 02 14:45:34.653361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653325    5034 cpu_manager.go:414] "containerMap: removing container" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerName="cleanup"
Nov 02 14:45:34.653361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653330    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerName="cleanup"
Nov 02 14:45:34.653361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653342    5034 cpu_manager.go:414] "containerMap: removing container" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerName="cleanup"
Nov 02 14:45:34.653361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653347    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerName="cleanup"
Nov 02 14:45:34.653361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653356    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerName="cleanup"
Nov 02 14:45:34.653361 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653361    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653370    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0e9989de-be08-4242-944b-9db03854c4ed" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653375    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9989de-be08-4242-944b-9db03854c4ed" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653384    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653388    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653393    5034 cpu_manager.go:414] "containerMap: removing container" podUID="df67ed0b-ff8d-4774-a628-3acbf17eb2db" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653398    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df67ed0b-ff8d-4774-a628-3acbf17eb2db" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653403    5034 cpu_manager.go:414] "containerMap: removing container" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653407    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653414    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653418    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653426    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653430    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653437    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653441    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653449    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653454    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653460    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653464    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653470    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653475    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653482    5034 cpu_manager.go:414] "containerMap: removing container" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653487    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653495    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653499    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653507    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653512    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653519    5034 cpu_manager.go:414] "containerMap: removing container" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653523    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653531    5034 cpu_manager.go:414] "containerMap: removing container" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653535    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653544    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653549    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653556    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653562    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653573    5034 cpu_manager.go:414] "containerMap: removing container" podUID="82af8d05-2c41-441f-b905-a1b186bfcd11" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653578    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82af8d05-2c41-441f-b905-a1b186bfcd11" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653586    5034 cpu_manager.go:414] "containerMap: removing container" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653590    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653601    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653606    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerName="cleanup"
Nov 02 14:45:34.653593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653614    5034 cpu_manager.go:414] "containerMap: removing container" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653618    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653628    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653634    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653646    5034 cpu_manager.go:414] "containerMap: removing container" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653653    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653665    5034 cpu_manager.go:414] "containerMap: removing container" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653670    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653678    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653682    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653688    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653692    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653700    5034 cpu_manager.go:414] "containerMap: removing container" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653705    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653710    5034 cpu_manager.go:414] "containerMap: removing container" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653714    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653725    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ac656679-7557-495a-a88e-8b166be2da0e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653729    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac656679-7557-495a-a88e-8b166be2da0e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653736    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653740    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653747    5034 cpu_manager.go:414] "containerMap: removing container" podUID="82af8d05-2c41-441f-b905-a1b186bfcd11" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653751    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82af8d05-2c41-441f-b905-a1b186bfcd11" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653759    5034 cpu_manager.go:414] "containerMap: removing container" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653763    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653773    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653793    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653800    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653805    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653813    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653817    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653826    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653829    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653837    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653841    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653850    5034 cpu_manager.go:414] "containerMap: removing container" podUID="df67ed0b-ff8d-4774-a628-3acbf17eb2db" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653854    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df67ed0b-ff8d-4774-a628-3acbf17eb2db" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653860    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653864    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653872    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653876    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653883    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653887    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653897    5034 cpu_manager.go:414] "containerMap: removing container" podUID="de00d3b1-f2d5-46c7-a29c-fb1576200196" containerName="payload-processing"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653901    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de00d3b1-f2d5-46c7-a29c-fb1576200196" containerName="payload-processing"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653909    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653914    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653922    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0e9989de-be08-4242-944b-9db03854c4ed" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653926    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9989de-be08-4242-944b-9db03854c4ed" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653933    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653937    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653944    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653949    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653959    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653963    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653971    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653975    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653986    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653990    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.653998    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ac656679-7557-495a-a88e-8b166be2da0e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654003    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac656679-7557-495a-a88e-8b166be2da0e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654012    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654017    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654024    5034 cpu_manager.go:414] "containerMap: removing container" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654028    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654036    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ac656679-7557-495a-a88e-8b166be2da0e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654040    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac656679-7557-495a-a88e-8b166be2da0e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654051    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654055    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654062    5034 cpu_manager.go:414] "containerMap: removing container" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654067    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654217    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="de00d3b1-f2d5-46c7-a29c-fb1576200196" containerName="payload-processing"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654230    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654236    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9989de-be08-4242-944b-9db03854c4ed" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654243    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654249    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654254    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654258    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654264    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654270    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654276    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac656679-7557-495a-a88e-8b166be2da0e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654281    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654286    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654292    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654298    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654304    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac656679-7557-495a-a88e-8b166be2da0e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654309    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654314    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654320    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654327    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654332    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654340    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654346    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654352    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9989de-be08-4242-944b-9db03854c4ed" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654357    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654362    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="df67ed0b-ff8d-4774-a628-3acbf17eb2db" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654366    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654374    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654379    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654389    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654393    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654413    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654420    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654428    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654435    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654442    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654449    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654455    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654460    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654470    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b62861-322e-48a0-938d-539acb06cbcf" containerName="istio-proxy"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654476    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="82af8d05-2c41-441f-b905-a1b186bfcd11" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654482    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654487    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654495    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654501    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654507    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="708b38e0-5f4a-4669-84e7-f74b02a3425e" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654514    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b44b24-e712-4d2e-9a1d-9c219925ff44" containerName="istio-proxy"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654519    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654524    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="65501c91-8e19-4ee7-bb4c-ec746643d814" containerName="istio-proxy"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654529    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654536    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654542    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654549    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="82af8d05-2c41-441f-b905-a1b186bfcd11" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654555    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="de34363d-7a31-4d79-8471-9b4e69d181d5" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654562    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654571    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654579    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="df67ed0b-ff8d-4774-a628-3acbf17eb2db" containerName="cleanup"
Nov 02 14:45:34.655052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.654590    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerName="cleanup"
Nov 02 14:45:34.659787 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.658011    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.679040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.678994    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"]
Nov 02 14:45:34.685440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.685372    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:45:34.685901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.685880    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:45:34.686171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.686152    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:45:34.688678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.688654    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:45:34.767459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.767426    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6hq\" (UniqueName: \"kubernetes.io/projected/9616290c-1ed2-450d-8532-525e9d9eb7fd-kube-api-access-pt6hq\") pod \"maas-api-revoke-sub-54fe25ac-rdgpq\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.767610 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.767494    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54fe25ac-rdgpq\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.767698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.767677    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54fe25ac-rdgpq\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.869157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.869127    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54fe25ac-rdgpq\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.869321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.869277    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54fe25ac-rdgpq\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.869510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.869457    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6hq\" (UniqueName: \"kubernetes.io/projected/9616290c-1ed2-450d-8532-525e9d9eb7fd-kube-api-access-pt6hq\") pod \"maas-api-revoke-sub-54fe25ac-rdgpq\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.869992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.869966    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54fe25ac-rdgpq\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.871637 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.871613    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54fe25ac-rdgpq\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.897193 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.897171    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6hq\" (UniqueName: \"kubernetes.io/projected/9616290c-1ed2-450d-8532-525e9d9eb7fd-kube-api-access-pt6hq\") pod \"maas-api-revoke-sub-54fe25ac-rdgpq\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:34.985402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:34.985376    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:35.348944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:35.348912    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"]
Nov 02 14:45:35.349573 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:35.349549    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9616290c_1ed2_450d_8532_525e9d9eb7fd.slice/crio-cf370390ee61feb60d6fcfea864b011110e1ae9ba2203d00facecb63f9ec5be8 WatchSource:0}: Error finding container cf370390ee61feb60d6fcfea864b011110e1ae9ba2203d00facecb63f9ec5be8: Status 404 returned error can't find the container with id cf370390ee61feb60d6fcfea864b011110e1ae9ba2203d00facecb63f9ec5be8
Nov 02 14:45:36.193057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:36.192963    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj" podUID="1c842308-c4ab-4068-bd2f-e1106361cb16" containerName="payload-processing" containerID="cri-o://d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447" gracePeriod=25
Nov 02 14:45:36.348321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:36.348284    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq" event={"ID":"9616290c-1ed2-450d-8532-525e9d9eb7fd","Type":"ContainerStarted","Data":"3b49b59fde8558b0faf734b520789db6260e0ab08e7ad65c4264ea29e3b562b0"}
Nov 02 14:45:36.348611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:36.348577    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq" event={"ID":"9616290c-1ed2-450d-8532-525e9d9eb7fd","Type":"ContainerStarted","Data":"cf370390ee61feb60d6fcfea864b011110e1ae9ba2203d00facecb63f9ec5be8"}
Nov 02 14:45:36.395742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:36.395675    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq" podStartSLOduration=2.395652 podStartE2EDuration="2.395652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:36.39229181 +0000 UTC m=+3727.143596249" watchObservedRunningTime="2026-11-02 14:45:36.395652 +0000 UTC m=+3727.146956490"
Nov 02 14:45:37.362170 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:37.362130    5034 generic.go:356] "Generic (PLEG): container finished" podID="9616290c-1ed2-450d-8532-525e9d9eb7fd" containerID="3b49b59fde8558b0faf734b520789db6260e0ab08e7ad65c4264ea29e3b562b0" exitCode=0
Nov 02 14:45:37.362572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:37.362169    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq" event={"ID":"9616290c-1ed2-450d-8532-525e9d9eb7fd","Type":"ContainerDied","Data":"3b49b59fde8558b0faf734b520789db6260e0ab08e7ad65c4264ea29e3b562b0"}
Nov 02 14:45:38.585719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.585698    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:38.746234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.746210    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-xks-ca\") pod \"9616290c-1ed2-450d-8532-525e9d9eb7fd\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") "
Nov 02 14:45:38.746329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.746286    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9616290c-1ed2-450d-8532-525e9d9eb7fd-kube-api-access-pt6hq\" (UniqueName: \"kubernetes.io/projected/9616290c-1ed2-450d-8532-525e9d9eb7fd-kube-api-access-pt6hq\") pod \"9616290c-1ed2-450d-8532-525e9d9eb7fd\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") "
Nov 02 14:45:38.746410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.746386    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-openshift-ca\") pod \"9616290c-1ed2-450d-8532-525e9d9eb7fd\" (UID: \"9616290c-1ed2-450d-8532-525e9d9eb7fd\") "
Nov 02 14:45:38.746730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.746703    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-openshift-ca" pod "9616290c-1ed2-450d-8532-525e9d9eb7fd" (UID: "9616290c-1ed2-450d-8532-525e9d9eb7fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:38.747412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.747389    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:38.748445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.748421    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-xks-ca" pod "9616290c-1ed2-450d-8532-525e9d9eb7fd" (UID: "9616290c-1ed2-450d-8532-525e9d9eb7fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:45:38.748518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.748460    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9616290c-1ed2-450d-8532-525e9d9eb7fd-kube-api-access-pt6hq" pod "9616290c-1ed2-450d-8532-525e9d9eb7fd" (UID: "9616290c-1ed2-450d-8532-525e9d9eb7fd"). InnerVolumeSpecName "kube-api-access-pt6hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:38.849230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.849202    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9616290c-1ed2-450d-8532-525e9d9eb7fd-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:38.849230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:38.849226    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt6hq\" (UniqueName: \"kubernetes.io/projected/9616290c-1ed2-450d-8532-525e9d9eb7fd-kube-api-access-pt6hq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:39.381757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:39.381731    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"
Nov 02 14:45:39.381990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:39.381774    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq" event={"ID":"9616290c-1ed2-450d-8532-525e9d9eb7fd","Type":"ContainerDied","Data":"cf370390ee61feb60d6fcfea864b011110e1ae9ba2203d00facecb63f9ec5be8"}
Nov 02 14:45:39.381990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:39.381860    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf370390ee61feb60d6fcfea864b011110e1ae9ba2203d00facecb63f9ec5be8"
Nov 02 14:45:40.140764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:40.140727    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj" podUID="1c842308-c4ab-4068-bd2f-e1106361cb16" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:45:46.107406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.107371    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"]
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108671    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108692    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dcb0f45-1ec6-43ac-a9af-571593753a63" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108714    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108721    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108740    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108748    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108766    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0e9989de-be08-4242-944b-9db03854c4ed" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108773    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e9989de-be08-4242-944b-9db03854c4ed" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108820    5034 cpu_manager.go:414] "containerMap: removing container" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108827    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108849    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108856    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108874    5034 cpu_manager.go:414] "containerMap: removing container" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108880    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f4abb8-f88b-42df-83c4-0dbffb278927" containerName="cleanup"
Nov 02 14:45:46.108890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108898    5034 cpu_manager.go:414] "containerMap: removing container" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108905    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108929    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108936    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b296f5f-1030-4705-998d-4459d0706cb0" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108950    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108956    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108970    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108977    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108988    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.108995    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109005    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9616290c-1ed2-450d-8532-525e9d9eb7fd" containerName="revoke-keys"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109012    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9616290c-1ed2-450d-8532-525e9d9eb7fd" containerName="revoke-keys"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109042    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109049    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3eed1b-a630-4ee7-b3fc-de9d967c544d" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109248    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a9e4f4-79f9-4e98-be5e-f871f11b183d" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109264    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac656679-7557-495a-a88e-8b166be2da0e" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109275    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3238cdd3-2e70-456b-afe8-6cdbf2b20b89" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109293    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6518f1f5-1243-49f7-8bf2-b174142e754a" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109302    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e9989de-be08-4242-944b-9db03854c4ed" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109313    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bfda5e-d485-4a81-835b-4c369b3666e4" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109326    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ee5b3f-068d-40b7-aaf1-461643ff7fe7" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109335    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5527c983-4d3e-4d15-8118-3f4f656b8051" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109343    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29378f6-902d-4efb-977a-dd1f13dd5248" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109355    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9616290c-1ed2-450d-8532-525e9d9eb7fd" containerName="revoke-keys"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109369    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d83cf56-9c22-47e5-aef4-087b779a2f65" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109381    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdbb368-93f7-4f30-89ef-881ec3528d6c" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109395    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b407ad8b-5856-4f5f-baf7-baebc7fe9233" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109407    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab9ef8a-3eb0-4899-ab01-3f50e9ed171f" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109416    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd77d21c-5627-41d7-ab81-ec7a4651f524" containerName="cleanup"
Nov 02 14:45:46.109462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.109426    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63a0f7e-2b21-46b1-80d2-61d439daf047" containerName="cleanup"
Nov 02 14:45:46.113368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.113343    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.128166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.128143    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"]
Nov 02 14:45:46.136731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.136572    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:45:46.136731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.136596    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:45:46.136731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.136650    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:45:46.143387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.143368    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:45:46.164169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.164147    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stvnr\" (UniqueName: \"kubernetes.io/projected/bc418012-6613-46f8-b646-afb5cd9a6bca-kube-api-access-stvnr\") pod \"maas-api-revoke-sub-1045bfbd-kx9pj\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.164364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.164346    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1045bfbd-kx9pj\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.164442 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.164403    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1045bfbd-kx9pj\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.265286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.265262    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stvnr\" (UniqueName: \"kubernetes.io/projected/bc418012-6613-46f8-b646-afb5cd9a6bca-kube-api-access-stvnr\") pod \"maas-api-revoke-sub-1045bfbd-kx9pj\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.265413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.265385    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1045bfbd-kx9pj\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.265470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.265424    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1045bfbd-kx9pj\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.266109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.266085    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1045bfbd-kx9pj\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.267904 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.267886    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1045bfbd-kx9pj\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.290376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.290347    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stvnr\" (UniqueName: \"kubernetes.io/projected/bc418012-6613-46f8-b646-afb5cd9a6bca-kube-api-access-stvnr\") pod \"maas-api-revoke-sub-1045bfbd-kx9pj\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.428677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.428612    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:46.738710 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:45:46.738672    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc418012_6613_46f8_b646_afb5cd9a6bca.slice/crio-1bb8bffe502b73d3cd21cc18530611ae82fa0304d32e7f23540c180ed70eed81 WatchSource:0}: Error finding container 1bb8bffe502b73d3cd21cc18530611ae82fa0304d32e7f23540c180ed70eed81: Status 404 returned error can't find the container with id 1bb8bffe502b73d3cd21cc18530611ae82fa0304d32e7f23540c180ed70eed81
Nov 02 14:45:46.741113 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:46.741090    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"]
Nov 02 14:45:47.454022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:47.453940    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj" event={"ID":"bc418012-6613-46f8-b646-afb5cd9a6bca","Type":"ContainerStarted","Data":"1b05e9c5cb8bd5b5bcdaf2bde4ea1efa31fc597dd62352341d1c4283103ab8d4"}
Nov 02 14:45:47.454022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:47.453987    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj" event={"ID":"bc418012-6613-46f8-b646-afb5cd9a6bca","Type":"ContainerStarted","Data":"1bb8bffe502b73d3cd21cc18530611ae82fa0304d32e7f23540c180ed70eed81"}
Nov 02 14:45:47.498720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:47.498670    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj" podStartSLOduration=1.498654765 podStartE2EDuration="1.498654765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:45:47.496957447 +0000 UTC m=+3738.248261882" watchObservedRunningTime="2026-11-02 14:45:47.498654765 +0000 UTC m=+3738.249959220"
Nov 02 14:45:48.415610 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:48.415566    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:45:48.465380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:48.465353    5034 generic.go:356] "Generic (PLEG): container finished" podID="bc418012-6613-46f8-b646-afb5cd9a6bca" containerID="1b05e9c5cb8bd5b5bcdaf2bde4ea1efa31fc597dd62352341d1c4283103ab8d4" exitCode=0
Nov 02 14:45:48.465735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:48.465409    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj" event={"ID":"bc418012-6613-46f8-b646-afb5cd9a6bca","Type":"ContainerDied","Data":"1b05e9c5cb8bd5b5bcdaf2bde4ea1efa31fc597dd62352341d1c4283103ab8d4"}
Nov 02 14:45:49.658787 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.658758    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:49.725884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.725801    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-openshift-ca\") pod \"bc418012-6613-46f8-b646-afb5cd9a6bca\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") "
Nov 02 14:45:49.725884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.725861    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-xks-ca\") pod \"bc418012-6613-46f8-b646-afb5cd9a6bca\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") "
Nov 02 14:45:49.726183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.725910    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc418012-6613-46f8-b646-afb5cd9a6bca-kube-api-access-stvnr\" (UniqueName: \"kubernetes.io/projected/bc418012-6613-46f8-b646-afb5cd9a6bca-kube-api-access-stvnr\") pod \"bc418012-6613-46f8-b646-afb5cd9a6bca\" (UID: \"bc418012-6613-46f8-b646-afb5cd9a6bca\") "
Nov 02 14:45:49.726259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.726188    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-openshift-ca" pod "bc418012-6613-46f8-b646-afb5cd9a6bca" (UID: "bc418012-6613-46f8-b646-afb5cd9a6bca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:49.726914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.726885    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:49.728155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.728130    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-xks-ca" pod "bc418012-6613-46f8-b646-afb5cd9a6bca" (UID: "bc418012-6613-46f8-b646-afb5cd9a6bca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:45:49.728244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.728226    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc418012-6613-46f8-b646-afb5cd9a6bca-kube-api-access-stvnr" pod "bc418012-6613-46f8-b646-afb5cd9a6bca" (UID: "bc418012-6613-46f8-b646-afb5cd9a6bca"). InnerVolumeSpecName "kube-api-access-stvnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:49.828196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.828156    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc418012-6613-46f8-b646-afb5cd9a6bca-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:49.828196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:49.828189    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stvnr\" (UniqueName: \"kubernetes.io/projected/bc418012-6613-46f8-b646-afb5cd9a6bca-kube-api-access-stvnr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:50.141421 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:50.141382    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj" podUID="1c842308-c4ab-4068-bd2f-e1106361cb16" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:45:50.486644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:50.486608    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj" event={"ID":"bc418012-6613-46f8-b646-afb5cd9a6bca","Type":"ContainerDied","Data":"1bb8bffe502b73d3cd21cc18530611ae82fa0304d32e7f23540c180ed70eed81"}
Nov 02 14:45:50.486644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:50.486640    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"
Nov 02 14:45:50.486951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:50.486659    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bb8bffe502b73d3cd21cc18530611ae82fa0304d32e7f23540c180ed70eed81"
Nov 02 14:45:56.493273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.493244    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:45:56.561666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.561626    5034 generic.go:356] "Generic (PLEG): container finished" podID="1c842308-c4ab-4068-bd2f-e1106361cb16" containerID="d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447" exitCode=0
Nov 02 14:45:56.562261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.561674    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"
Nov 02 14:45:56.562826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.561691    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj" event={"ID":"1c842308-c4ab-4068-bd2f-e1106361cb16","Type":"ContainerDied","Data":"d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447"}
Nov 02 14:45:56.563018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.562997    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj" event={"ID":"1c842308-c4ab-4068-bd2f-e1106361cb16","Type":"ContainerDied","Data":"140cc9f99fa499cd0c28dc060a6dd5545eaf8b7de3c093858916260b122ea332"}
Nov 02 14:45:56.563171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.563156    5034 scope.go:122] "RemoveContainer" containerID="d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447"
Nov 02 14:45:56.585587 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.585563    5034 scope.go:122] "RemoveContainer" containerID="d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447"
Nov 02 14:45:56.585998 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:45:56.585972    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447\": container with ID starting with d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447 not found: ID does not exist" containerID="d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447"
Nov 02 14:45:56.586083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.586008    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447"} err="failed to get container status \"d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447\": rpc error: code = NotFound desc = could not find container \"d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447\": container with ID starting with d61a362b21a5af34baba8c954b8fc8182d75f503be1023f49c0af7108d765447 not found: ID does not exist"
Nov 02 14:45:56.650491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.650454    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c842308-c4ab-4068-bd2f-e1106361cb16-kube-api-access-xf2xf\" (UniqueName: \"kubernetes.io/projected/1c842308-c4ab-4068-bd2f-e1106361cb16-kube-api-access-xf2xf\") pod \"1c842308-c4ab-4068-bd2f-e1106361cb16\" (UID: \"1c842308-c4ab-4068-bd2f-e1106361cb16\") "
Nov 02 14:45:56.650856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.650836    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c842308-c4ab-4068-bd2f-e1106361cb16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c842308-c4ab-4068-bd2f-e1106361cb16-plugins-config-volume\") pod \"1c842308-c4ab-4068-bd2f-e1106361cb16\" (UID: \"1c842308-c4ab-4068-bd2f-e1106361cb16\") "
Nov 02 14:45:56.651221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.651195    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c842308-c4ab-4068-bd2f-e1106361cb16-plugins-config-volume" pod "1c842308-c4ab-4068-bd2f-e1106361cb16" (UID: "1c842308-c4ab-4068-bd2f-e1106361cb16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:45:56.651459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.651440    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c842308-c4ab-4068-bd2f-e1106361cb16-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:56.652650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.652628    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c842308-c4ab-4068-bd2f-e1106361cb16-kube-api-access-xf2xf" pod "1c842308-c4ab-4068-bd2f-e1106361cb16" (UID: "1c842308-c4ab-4068-bd2f-e1106361cb16"). InnerVolumeSpecName "kube-api-access-xf2xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:45:56.753371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.753339    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf2xf\" (UniqueName: \"kubernetes.io/projected/1c842308-c4ab-4068-bd2f-e1106361cb16-kube-api-access-xf2xf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:45:56.916195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.916148    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"]
Nov 02 14:45:56.921324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:56.921295    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-e1136e-65885fc69c-85hkj"]
Nov 02 14:45:57.397317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:45:57.397288    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c842308-c4ab-4068-bd2f-e1106361cb16" path="/var/lib/kubelet/pods/1c842308-c4ab-4068-bd2f-e1106361cb16/volumes"
Nov 02 14:46:14.401054 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:14.401011    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"]
Nov 02 14:46:18.421107 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:46:18.421070    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:46:18.730334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:18.730258    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"]
Nov 02 14:46:18.730493 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:18.730458    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy" containerID="cri-o://fedc9bd63df4d6140957b55ffc34a61bdb36e023d76138716a98aa22a626d7a5" gracePeriod=30
Nov 02 14:46:19.401579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.401547    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97" podUID="5fb6543b-ac5e-49cc-845e-35c0b1d783d3" containerName="payload-processing" containerID="cri-o://31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046" gracePeriod=25
Nov 02 14:46:19.730041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.730013    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:46:19.810642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.810581    5034 generic.go:356] "Generic (PLEG): container finished" podID="5fb6543b-ac5e-49cc-845e-35c0b1d783d3" containerID="31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046" exitCode=0
Nov 02 14:46:19.810642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.810631    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"
Nov 02 14:46:19.810942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.810702    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97" event={"ID":"5fb6543b-ac5e-49cc-845e-35c0b1d783d3","Type":"ContainerDied","Data":"31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046"}
Nov 02 14:46:19.810942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.810825    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97" event={"ID":"5fb6543b-ac5e-49cc-845e-35c0b1d783d3","Type":"ContainerDied","Data":"9c839ba28a3b9ad8f5ab81351f8720231e1d7c1556f7635a0567c8d6c4635b54"}
Nov 02 14:46:19.810942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.810865    5034 scope.go:122] "RemoveContainer" containerID="31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046"
Nov 02 14:46:19.824567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.824547    5034 scope.go:122] "RemoveContainer" containerID="31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046"
Nov 02 14:46:19.824907 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:46:19.824882    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046\": container with ID starting with 31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046 not found: ID does not exist" containerID="31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046"
Nov 02 14:46:19.825021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.824913    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046"} err="failed to get container status \"31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046\": rpc error: code = NotFound desc = could not find container \"31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046\": container with ID starting with 31b4c4f3c13dd9c2dae450247337b745447fc481a88de570ca67e5d9d3e23046 not found: ID does not exist"
Nov 02 14:46:19.862723 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.862698    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-plugins-config-volume\") pod \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\" (UID: \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\") "
Nov 02 14:46:19.862913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.862862    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-kube-api-access-tcwtg\" (UniqueName: \"kubernetes.io/projected/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-kube-api-access-tcwtg\") pod \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\" (UID: \"5fb6543b-ac5e-49cc-845e-35c0b1d783d3\") "
Nov 02 14:46:19.863117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.863094    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-plugins-config-volume" pod "5fb6543b-ac5e-49cc-845e-35c0b1d783d3" (UID: "5fb6543b-ac5e-49cc-845e-35c0b1d783d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:19.863426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.863407    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:19.864813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.864771    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-kube-api-access-tcwtg" pod "5fb6543b-ac5e-49cc-845e-35c0b1d783d3" (UID: "5fb6543b-ac5e-49cc-845e-35c0b1d783d3"). InnerVolumeSpecName "kube-api-access-tcwtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:19.965410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:19.965338    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcwtg\" (UniqueName: \"kubernetes.io/projected/5fb6543b-ac5e-49cc-845e-35c0b1d783d3-kube-api-access-tcwtg\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:20.167743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:20.167707    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"]
Nov 02 14:46:20.180654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:20.180624    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fe1881-56fc7db7f-vmc97"]
Nov 02 14:46:21.399183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:21.399148    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb6543b-ac5e-49cc-845e-35c0b1d783d3" path="/var/lib/kubelet/pods/5fb6543b-ac5e-49cc-845e-35c0b1d783d3/volumes"
Nov 02 14:46:22.692288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:22.692241    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"]
Nov 02 14:46:22.692818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:22.692504    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy" containerID="cri-o://e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b" gracePeriod=30
Nov 02 14:46:23.966596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:23.966543    5034 patch_prober.go:29] interesting pod/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:46:23.967085 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:23.966667    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:46:24.015614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.015580    5034 generic.go:356] "Generic (PLEG): container finished" podID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerID="fedc9bd63df4d6140957b55ffc34a61bdb36e023d76138716a98aa22a626d7a5" exitCode=0
Nov 02 14:46:24.015810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.015658    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" event={"ID":"48fdf194-1d79-4618-a09b-2fc9c16d94eb","Type":"ContainerDied","Data":"fedc9bd63df4d6140957b55ffc34a61bdb36e023d76138716a98aa22a626d7a5"}
Nov 02 14:46:24.050718 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.050688    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.182109    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-token\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-token\") pod \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") "
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.182176    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-kube-api-access-hl8hv\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-kube-api-access-hl8hv\") pod \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") "
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.182208    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-certs\") pod \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") "
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.182256    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-credential-socket\") pod \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") "
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.182344    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-envoy\") pod \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") "
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.182499    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istiod-ca-cert\") pod \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") "
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.182629    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-data\") pod \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") "
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.182673    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-socket\") pod \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") "
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.182734    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-podinfo\") pod \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\" (UID: \"48fdf194-1d79-4618-a09b-2fc9c16d94eb\") "
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.183989    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-credential-socket" pod "48fdf194-1d79-4618-a09b-2fc9c16d94eb" (UID: "48fdf194-1d79-4618-a09b-2fc9c16d94eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.184205    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-certs" pod "48fdf194-1d79-4618-a09b-2fc9c16d94eb" (UID: "48fdf194-1d79-4618-a09b-2fc9c16d94eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.184814    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istiod-ca-cert" pod "48fdf194-1d79-4618-a09b-2fc9c16d94eb" (UID: "48fdf194-1d79-4618-a09b-2fc9c16d94eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.185027    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-data" pod "48fdf194-1d79-4618-a09b-2fc9c16d94eb" (UID: "48fdf194-1d79-4618-a09b-2fc9c16d94eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.185073    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-socket" pod "48fdf194-1d79-4618-a09b-2fc9c16d94eb" (UID: "48fdf194-1d79-4618-a09b-2fc9c16d94eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.186028    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-kube-api-access-hl8hv" pod "48fdf194-1d79-4618-a09b-2fc9c16d94eb" (UID: "48fdf194-1d79-4618-a09b-2fc9c16d94eb"). InnerVolumeSpecName "kube-api-access-hl8hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:24.186824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.186699    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-podinfo" pod "48fdf194-1d79-4618-a09b-2fc9c16d94eb" (UID: "48fdf194-1d79-4618-a09b-2fc9c16d94eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:46:24.187527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.186985    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-token" pod "48fdf194-1d79-4618-a09b-2fc9c16d94eb" (UID: "48fdf194-1d79-4618-a09b-2fc9c16d94eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:24.188392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.188370    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-envoy" pod "48fdf194-1d79-4618-a09b-2fc9c16d94eb" (UID: "48fdf194-1d79-4618-a09b-2fc9c16d94eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:24.285504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.285422    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:24.285504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.285455    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl8hv\" (UniqueName: \"kubernetes.io/projected/48fdf194-1d79-4618-a09b-2fc9c16d94eb-kube-api-access-hl8hv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:24.285504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.285466    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:24.285504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.285474    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:24.285504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.285502    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:24.285504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.285511    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:24.285925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.285519    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:24.285925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.285528    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48fdf194-1d79-4618-a09b-2fc9c16d94eb-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:24.285925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:24.285535    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48fdf194-1d79-4618-a09b-2fc9c16d94eb-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:25.026230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:25.026201    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"
Nov 02 14:46:25.026658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:25.026201    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d" event={"ID":"48fdf194-1d79-4618-a09b-2fc9c16d94eb","Type":"ContainerDied","Data":"0aee75176f912173e739ff7b40a84989ff3d7666afdcfc14f95a4dc340157947"}
Nov 02 14:46:25.026658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:25.026335    5034 scope.go:122] "RemoveContainer" containerID="fedc9bd63df4d6140957b55ffc34a61bdb36e023d76138716a98aa22a626d7a5"
Nov 02 14:46:25.098818 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:25.098764    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"]
Nov 02 14:46:25.115110 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:25.115080    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1a562c-openshift-default-776b45d745-d6f7d"]
Nov 02 14:46:25.398432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:25.398358    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" path="/var/lib/kubelet/pods/48fdf194-1d79-4618-a09b-2fc9c16d94eb/volumes"
Nov 02 14:46:27.244913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:27.244879    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"]
Nov 02 14:46:27.245411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:27.245122    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy" containerID="cri-o://e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142" gracePeriod=30
Nov 02 14:46:28.018880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.018854    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:46:28.045268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045231    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-envoy\") pod \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") "
Nov 02 14:46:28.045419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045300    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istiod-ca-cert\") pod \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") "
Nov 02 14:46:28.045419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045344    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-podinfo\") pod \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") "
Nov 02 14:46:28.045558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045461    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-certs\") pod \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") "
Nov 02 14:46:28.045558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045500    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-data\") pod \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") "
Nov 02 14:46:28.045558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045542    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-token\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-token\") pod \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") "
Nov 02 14:46:28.045719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045604    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-kube-api-access-5sdmz\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-kube-api-access-5sdmz\") pod \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") "
Nov 02 14:46:28.045719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045677    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istiod-ca-cert" pod "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" (UID: "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:28.045858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045747    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-credential-socket\") pod \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") "
Nov 02 14:46:28.045858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.045790    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-socket\") pod \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\" (UID: \"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979\") "
Nov 02 14:46:28.046681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.046143    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-credential-socket" pod "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" (UID: "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:28.046681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.046374    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-socket" pod "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" (UID: "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:28.046681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.046567    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-data" pod "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" (UID: "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:28.046681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.046620    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:28.046681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.046644    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:28.046681 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.046660    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:28.047236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.047209    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-certs" pod "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" (UID: "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:28.048016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.047991    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-envoy" pod "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" (UID: "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:28.048547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.048446    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-token" pod "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" (UID: "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:28.048547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.048516    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-podinfo" pod "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" (UID: "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:46:28.049403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.049383    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-kube-api-access-5sdmz" pod "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" (UID: "ee48d5e3-cdfb-45e9-96ce-e5d75ff37979"). InnerVolumeSpecName "kube-api-access-5sdmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:28.054856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.054828    5034 generic.go:356] "Generic (PLEG): container finished" podID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerID="e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b" exitCode=0
Nov 02 14:46:28.054954 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.054915    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"
Nov 02 14:46:28.055034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.054940    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" event={"ID":"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979","Type":"ContainerDied","Data":"e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b"}
Nov 02 14:46:28.055034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.055005    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l" event={"ID":"ee48d5e3-cdfb-45e9-96ce-e5d75ff37979","Type":"ContainerDied","Data":"ce5a0ddb2421d04c8766170a2c8d167a6019ee9c9132b1f2223147fe470a1cff"}
Nov 02 14:46:28.055145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.055046    5034 scope.go:122] "RemoveContainer" containerID="e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b"
Nov 02 14:46:28.089453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.089434    5034 scope.go:122] "RemoveContainer" containerID="74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379"
Nov 02 14:46:28.102317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.102297    5034 scope.go:122] "RemoveContainer" containerID="e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b"
Nov 02 14:46:28.102626 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:46:28.102604    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b\": container with ID starting with e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b not found: ID does not exist" containerID="e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b"
Nov 02 14:46:28.102732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.102632    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b"} err="failed to get container status \"e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b\": rpc error: code = NotFound desc = could not find container \"e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b\": container with ID starting with e4a53cf51e08c380ff5d8ee09be4e92932505267f4256f842656edb06eab017b not found: ID does not exist"
Nov 02 14:46:28.102732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.102649    5034 scope.go:122] "RemoveContainer" containerID="74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379"
Nov 02 14:46:28.103004 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:46:28.102980    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379\": container with ID starting with 74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379 not found: ID does not exist" containerID="74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379"
Nov 02 14:46:28.103056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.103010    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379"} err="failed to get container status \"74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379\": rpc error: code = NotFound desc = could not find container \"74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379\": container with ID starting with 74ffd213ebb738c6a21c5a20aaad91b8f323fba99cadae52dc8a8333d917f379 not found: ID does not exist"
Nov 02 14:46:28.126810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.126750    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"]
Nov 02 14:46:28.137119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.137094    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e1136e-openshift-default-59d8f9c4c9-r6p9l"]
Nov 02 14:46:28.148374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.148346    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:28.148494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.148376    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:28.148494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.148392    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:28.148494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.148405    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:28.148494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.148417    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:28.148494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:28.148430    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sdmz\" (UniqueName: \"kubernetes.io/projected/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979-kube-api-access-5sdmz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:29.397091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:29.397061    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" path="/var/lib/kubelet/pods/ee48d5e3-cdfb-45e9-96ce-e5d75ff37979/volumes"
Nov 02 14:46:32.573203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.573175    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:46:32.620475 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620442    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-credential-socket\") pod \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") "
Nov 02 14:46:32.620653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620517    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istiod-ca-cert\") pod \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") "
Nov 02 14:46:32.620653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620566    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-podinfo\") pod \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") "
Nov 02 14:46:32.620653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620599    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-certs\") pod \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") "
Nov 02 14:46:32.620653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620630    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-kube-api-access-v7pwp\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-kube-api-access-v7pwp\") pod \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") "
Nov 02 14:46:32.620934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620670    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-envoy\") pod \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") "
Nov 02 14:46:32.620934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620717    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-credential-socket" pod "bfd9dc24-44e7-4c1e-ac79-e45383aaab73" (UID: "bfd9dc24-44e7-4c1e-ac79-e45383aaab73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:32.620934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620753    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-token\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-token\") pod \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") "
Nov 02 14:46:32.621093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620946    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-socket\") pod \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") "
Nov 02 14:46:32.621093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.620982    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-certs" pod "bfd9dc24-44e7-4c1e-ac79-e45383aaab73" (UID: "bfd9dc24-44e7-4c1e-ac79-e45383aaab73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:32.621093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.621007    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istiod-ca-cert" pod "bfd9dc24-44e7-4c1e-ac79-e45383aaab73" (UID: "bfd9dc24-44e7-4c1e-ac79-e45383aaab73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:32.621264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.621109    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-data\") pod \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\" (UID: \"bfd9dc24-44e7-4c1e-ac79-e45383aaab73\") "
Nov 02 14:46:32.621368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.621341    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-socket" pod "bfd9dc24-44e7-4c1e-ac79-e45383aaab73" (UID: "bfd9dc24-44e7-4c1e-ac79-e45383aaab73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:32.621438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.621382    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-data" pod "bfd9dc24-44e7-4c1e-ac79-e45383aaab73" (UID: "bfd9dc24-44e7-4c1e-ac79-e45383aaab73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:32.621806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.621768    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:32.621913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.621806    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:32.621913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.621821    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:32.621913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.621833    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:32.621913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.621845    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:32.622940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.622911    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-podinfo" pod "bfd9dc24-44e7-4c1e-ac79-e45383aaab73" (UID: "bfd9dc24-44e7-4c1e-ac79-e45383aaab73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:46:32.623305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.623279    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-envoy" pod "bfd9dc24-44e7-4c1e-ac79-e45383aaab73" (UID: "bfd9dc24-44e7-4c1e-ac79-e45383aaab73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:32.623430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.623332    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-token" pod "bfd9dc24-44e7-4c1e-ac79-e45383aaab73" (UID: "bfd9dc24-44e7-4c1e-ac79-e45383aaab73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:32.623500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.623444    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-kube-api-access-v7pwp" pod "bfd9dc24-44e7-4c1e-ac79-e45383aaab73" (UID: "bfd9dc24-44e7-4c1e-ac79-e45383aaab73"). InnerVolumeSpecName "kube-api-access-v7pwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:32.722810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.722758    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:32.722810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.722806    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7pwp\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-kube-api-access-v7pwp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:32.722810 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.722818    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:32.723060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:32.722827    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd9dc24-44e7-4c1e-ac79-e45383aaab73-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:33.127306 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.127274    5034 generic.go:356] "Generic (PLEG): container finished" podID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerID="e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142" exitCode=0
Nov 02 14:46:33.127511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.127340    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"
Nov 02 14:46:33.127511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.127335    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" event={"ID":"bfd9dc24-44e7-4c1e-ac79-e45383aaab73","Type":"ContainerDied","Data":"e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142"}
Nov 02 14:46:33.127511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.127502    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5" event={"ID":"bfd9dc24-44e7-4c1e-ac79-e45383aaab73","Type":"ContainerDied","Data":"bf9d20891b31cd2efe5578a50c970297a33d1afadbe90d0962347cd3d52099b5"}
Nov 02 14:46:33.127688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.127534    5034 scope.go:122] "RemoveContainer" containerID="e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142"
Nov 02 14:46:33.140915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.140882    5034 scope.go:122] "RemoveContainer" containerID="e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142"
Nov 02 14:46:33.141204 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:46:33.141183    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142\": container with ID starting with e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142 not found: ID does not exist" containerID="e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142"
Nov 02 14:46:33.141259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.141211    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142"} err="failed to get container status \"e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142\": rpc error: code = NotFound desc = could not find container \"e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142\": container with ID starting with e8cf12c62f9caed971caad4530c1a9cbb03219823ec28c2ee62716364018d142 not found: ID does not exist"
Nov 02 14:46:33.200585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.200561    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"]
Nov 02 14:46:33.214308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.214280    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d39fa6-openshift-default-7fdf9bc587-s8kr5"]
Nov 02 14:46:33.398205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:33.398122    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" path="/var/lib/kubelet/pods/bfd9dc24-44e7-4c1e-ac79-e45383aaab73/volumes"
Nov 02 14:46:35.609876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.609832    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"]
Nov 02 14:46:35.611078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611054    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bc418012-6613-46f8-b646-afb5cd9a6bca" containerName="revoke-keys"
Nov 02 14:46:35.611078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611078    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc418012-6613-46f8-b646-afb5cd9a6bca" containerName="revoke-keys"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611123    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611131    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611153    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5fb6543b-ac5e-49cc-845e-35c0b1d783d3" containerName="payload-processing"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611161    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb6543b-ac5e-49cc-845e-35c0b1d783d3" containerName="payload-processing"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611192    5034 cpu_manager.go:414] "containerMap: removing container" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611199    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611211    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611218    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611253    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1c842308-c4ab-4068-bd2f-e1106361cb16" containerName="payload-processing"
Nov 02 14:46:35.611256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611260    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c842308-c4ab-4068-bd2f-e1106361cb16" containerName="payload-processing"
Nov 02 14:46:35.611797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611446    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c842308-c4ab-4068-bd2f-e1106361cb16" containerName="payload-processing"
Nov 02 14:46:35.611797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611470    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc418012-6613-46f8-b646-afb5cd9a6bca" containerName="revoke-keys"
Nov 02 14:46:35.611797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611486    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy"
Nov 02 14:46:35.611797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611496    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb6543b-ac5e-49cc-845e-35c0b1d783d3" containerName="payload-processing"
Nov 02 14:46:35.611797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611504    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy"
Nov 02 14:46:35.611797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611516    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="48fdf194-1d79-4618-a09b-2fc9c16d94eb" containerName="istio-proxy"
Nov 02 14:46:35.611797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.611525    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd9dc24-44e7-4c1e-ac79-e45383aaab73" containerName="istio-proxy"
Nov 02 14:46:35.624213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.616285    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.648154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.648122    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"]
Nov 02 14:46:35.658043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.658008    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ea3e8f-openshift-default-dockercfg-5ngr8\""
Nov 02 14:46:35.763735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.763700    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-envoy\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.763920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.763849    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8pwm\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-kube-api-access-k8pwm\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.763920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.763887    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.764007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.763923    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-data\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.764007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.763940    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-istio-token\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.764126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.764102    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-credential-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.764171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.764160    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-certs\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.764221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.764210    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d157582e-3896-46f9-a053-010c45191ec3-istio-podinfo\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.764261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.764252    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d157582e-3896-46f9-a053-010c45191ec3-istiod-ca-cert\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865091    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pwm\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-kube-api-access-k8pwm\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865143    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865192    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-data\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865212    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-istio-token\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865276    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-credential-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865351    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-certs\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865395    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d157582e-3896-46f9-a053-010c45191ec3-istio-podinfo\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865462    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d157582e-3896-46f9-a053-010c45191ec3-istiod-ca-cert\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865541    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865586    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-envoy\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865584    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-data\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.865711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865691    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-credential-socket\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.866033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.865747    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-certs\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.866297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.866274    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d157582e-3896-46f9-a053-010c45191ec3-istiod-ca-cert\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.867842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.867823    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-envoy\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.868007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.867987    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d157582e-3896-46f9-a053-010c45191ec3-istio-podinfo\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.890505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.890474    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-istio-token\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.890666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.890644    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pwm\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-kube-api-access-k8pwm\") pod \"e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") " pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:35.938371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:35.938351    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:36.271138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:36.271094    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:46:36.271256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:36.271193    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:46:36.271256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:36.271234    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:46:36.273621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:36.273594    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"]
Nov 02 14:46:37.200240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:37.200207    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" event={"ID":"d157582e-3896-46f9-a053-010c45191ec3","Type":"ContainerStarted","Data":"f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9"}
Nov 02 14:46:37.200670 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:37.200267    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" event={"ID":"d157582e-3896-46f9-a053-010c45191ec3","Type":"ContainerStarted","Data":"81bd2045bf580163ca6f019c5502e7e370b477eba7a92a0a948b9e11f8c0de23"}
Nov 02 14:46:37.271166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:37.271114    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" podStartSLOduration=2.271096442 podStartE2EDuration="2.271096442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:46:37.263478826 +0000 UTC m=+3788.014783264" watchObservedRunningTime="2026-11-02 14:46:37.271096442 +0000 UTC m=+3788.022400884"
Nov 02 14:46:37.939267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:37.939237    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:37.940694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:37.940666    5034 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body=
Nov 02 14:46:37.940842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:37.940722    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused"
Nov 02 14:46:38.939504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:38.939467    5034 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body=
Nov 02 14:46:38.939949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:38.939530    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused"
Nov 02 14:46:39.939111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:39.939073    5034 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body=
Nov 02 14:46:39.939323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:39.939140    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused"
Nov 02 14:46:40.948266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:40.947435    5034 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body=
Nov 02 14:46:40.948266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:40.947495    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused"
Nov 02 14:46:41.939049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:41.939009    5034 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body=
Nov 02 14:46:41.939236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:41.939069    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused"
Nov 02 14:46:42.938987 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:42.938945    5034 patch_prober.go:29] interesting pod/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body=
Nov 02 14:46:42.939522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:42.939008    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused"
Nov 02 14:46:43.944677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:43.944646    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:43.945182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:43.945164    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:43.945937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:43.945911    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:46:46.833750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:46.833717    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"]
Nov 02 14:46:46.834229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:46.834022    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy" containerID="cri-o://4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa" gracePeriod=30
Nov 02 14:46:48.416925 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:46:48.416881    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:46:49.112440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.112396    5034 patch_prober.go:29] interesting pod/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:46:49.112644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.112471    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:46:49.158521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.158481    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"]
Nov 02 14:46:49.159474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.159451    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy"
Nov 02 14:46:49.159474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.159473    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee48d5e3-cdfb-45e9-96ce-e5d75ff37979" containerName="istio-proxy"
Nov 02 14:46:49.163633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.163614    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.196701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.196667    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:46:49.196927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.196806    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:46:49.196927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.196809    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:46:49.197070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.196940    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:46:49.206990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.206956    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"]
Nov 02 14:46:49.327619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.327576    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ac6bc3-dj45n\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.327819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.327731    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67l9h\" (UniqueName: \"kubernetes.io/projected/0a7ed36b-7186-452d-a777-a449b205ab66-kube-api-access-67l9h\") pod \"maas-api-revoke-sub-a1ac6bc3-dj45n\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.327945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.327911    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ac6bc3-dj45n\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.429350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.429252    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ac6bc3-dj45n\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.429864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.429477    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ac6bc3-dj45n\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.429864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.429585    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67l9h\" (UniqueName: \"kubernetes.io/projected/0a7ed36b-7186-452d-a777-a449b205ab66-kube-api-access-67l9h\") pod \"maas-api-revoke-sub-a1ac6bc3-dj45n\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.430135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.430113    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1ac6bc3-dj45n\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.434641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.434608    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1ac6bc3-dj45n\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.454601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.454570    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67l9h\" (UniqueName: \"kubernetes.io/projected/0a7ed36b-7186-452d-a777-a449b205ab66-kube-api-access-67l9h\") pod \"maas-api-revoke-sub-a1ac6bc3-dj45n\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.486614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.486581    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:49.840565 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:46:49.840531    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7ed36b_7186_452d_a777_a449b205ab66.slice/crio-3ad7005cd3de0ec08b2be77f39f3a9ae0e352c42300abde47253d2c43428019b WatchSource:0}: Error finding container 3ad7005cd3de0ec08b2be77f39f3a9ae0e352c42300abde47253d2c43428019b: Status 404 returned error can't find the container with id 3ad7005cd3de0ec08b2be77f39f3a9ae0e352c42300abde47253d2c43428019b
Nov 02 14:46:49.850136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:49.849839    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"]
Nov 02 14:46:50.376529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:50.376431    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n" event={"ID":"0a7ed36b-7186-452d-a777-a449b205ab66","Type":"ContainerStarted","Data":"3ad7005cd3de0ec08b2be77f39f3a9ae0e352c42300abde47253d2c43428019b"}
Nov 02 14:46:51.386915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:51.386882    5034 generic.go:356] "Generic (PLEG): container finished" podID="0a7ed36b-7186-452d-a777-a449b205ab66" containerID="66e1f4daef75ed576e9ffe6fab6a9f82521608a8a52980a91572dab15259d803" exitCode=0
Nov 02 14:46:51.398260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:51.398228    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n" event={"ID":"0a7ed36b-7186-452d-a777-a449b205ab66","Type":"ContainerDied","Data":"66e1f4daef75ed576e9ffe6fab6a9f82521608a8a52980a91572dab15259d803"}
Nov 02 14:46:52.158004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.157971    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:46:52.292598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.292563    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-socket\") pod \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") "
Nov 02 14:46:52.292975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.292668    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-data\") pod \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") "
Nov 02 14:46:52.292975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.292753    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istiod-ca-cert\") pod \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") "
Nov 02 14:46:52.292975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.292842    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-podinfo\") pod \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") "
Nov 02 14:46:52.292975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.292886    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-certs\") pod \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") "
Nov 02 14:46:52.292975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.292888    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-socket" pod "2f1bf78b-d6af-48b8-b6c7-44c8f4694240" (UID: "2f1bf78b-d6af-48b8-b6c7-44c8f4694240"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:52.292975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.292930    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-credential-socket\") pod \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") "
Nov 02 14:46:52.293385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293036    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-token\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-token\") pod \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") "
Nov 02 14:46:52.293385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293080    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-kube-api-access-tzxc9\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-kube-api-access-tzxc9\") pod \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") "
Nov 02 14:46:52.293385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293111    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-data" pod "2f1bf78b-d6af-48b8-b6c7-44c8f4694240" (UID: "2f1bf78b-d6af-48b8-b6c7-44c8f4694240"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:52.293385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293169    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-envoy\") pod \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\" (UID: \"2f1bf78b-d6af-48b8-b6c7-44c8f4694240\") "
Nov 02 14:46:52.293385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293238    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istiod-ca-cert" pod "2f1bf78b-d6af-48b8-b6c7-44c8f4694240" (UID: "2f1bf78b-d6af-48b8-b6c7-44c8f4694240"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:52.293385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293351    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-certs" pod "2f1bf78b-d6af-48b8-b6c7-44c8f4694240" (UID: "2f1bf78b-d6af-48b8-b6c7-44c8f4694240"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:52.293927 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293905    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.294019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293931    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.294019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293948    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.294019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.293963    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.295134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.295105    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-credential-socket" pod "2f1bf78b-d6af-48b8-b6c7-44c8f4694240" (UID: "2f1bf78b-d6af-48b8-b6c7-44c8f4694240"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:52.295763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.295737    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-token" pod "2f1bf78b-d6af-48b8-b6c7-44c8f4694240" (UID: "2f1bf78b-d6af-48b8-b6c7-44c8f4694240"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:52.296104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.296081    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-kube-api-access-tzxc9" pod "2f1bf78b-d6af-48b8-b6c7-44c8f4694240" (UID: "2f1bf78b-d6af-48b8-b6c7-44c8f4694240"). InnerVolumeSpecName "kube-api-access-tzxc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:52.296293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.296270    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-envoy" pod "2f1bf78b-d6af-48b8-b6c7-44c8f4694240" (UID: "2f1bf78b-d6af-48b8-b6c7-44c8f4694240"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:46:52.296380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.296302    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-podinfo" pod "2f1bf78b-d6af-48b8-b6c7-44c8f4694240" (UID: "2f1bf78b-d6af-48b8-b6c7-44c8f4694240"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:46:52.395425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.395334    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.395425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.395365    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.395425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.395378    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.395425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.395390    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzxc9\" (UniqueName: \"kubernetes.io/projected/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-kube-api-access-tzxc9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.395425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.395404    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1bf78b-d6af-48b8-b6c7-44c8f4694240-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.398626 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.398593    5034 generic.go:356] "Generic (PLEG): container finished" podID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerID="4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa" exitCode=0
Nov 02 14:46:52.398799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.398663    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"
Nov 02 14:46:52.398799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.398681    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" event={"ID":"2f1bf78b-d6af-48b8-b6c7-44c8f4694240","Type":"ContainerDied","Data":"4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa"}
Nov 02 14:46:52.398799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.398746    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b" event={"ID":"2f1bf78b-d6af-48b8-b6c7-44c8f4694240","Type":"ContainerDied","Data":"350ddf995e46630c627dfd8d92ebe33f73ddb2cf8e4ecaa48b5e0533d614ea0e"}
Nov 02 14:46:52.398799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.398774    5034 scope.go:122] "RemoveContainer" containerID="4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa"
Nov 02 14:46:52.413868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.413845    5034 scope.go:122] "RemoveContainer" containerID="116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee"
Nov 02 14:46:52.425540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.425515    5034 scope.go:122] "RemoveContainer" containerID="4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa"
Nov 02 14:46:52.425945 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:46:52.425921    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa\": container with ID starting with 4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa not found: ID does not exist" containerID="4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa"
Nov 02 14:46:52.426062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.425952    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa"} err="failed to get container status \"4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa\": rpc error: code = NotFound desc = could not find container \"4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa\": container with ID starting with 4362d54855e98e8ff1c8ad6594e202c2e5bfacd090b6ec98b866e1a0a10575aa not found: ID does not exist"
Nov 02 14:46:52.426062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.425970    5034 scope.go:122] "RemoveContainer" containerID="116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee"
Nov 02 14:46:52.426266 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:46:52.426246    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee\": container with ID starting with 116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee not found: ID does not exist" containerID="116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee"
Nov 02 14:46:52.426310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.426271    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee"} err="failed to get container status \"116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee\": rpc error: code = NotFound desc = could not find container \"116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee\": container with ID starting with 116a48a0bb00b6fbecfdf2017c21f7248d86449b25b75956490cc91c52980aee not found: ID does not exist"
Nov 02 14:46:52.461264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.461233    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"]
Nov 02 14:46:52.469973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.469945    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fe1881-openshift-default-6794b48f5c-cz24b"]
Nov 02 14:46:52.567107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.567082    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:52.700605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.699923    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-xks-ca\") pod \"0a7ed36b-7186-452d-a777-a449b205ab66\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") "
Nov 02 14:46:52.700605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.700013    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-openshift-ca\") pod \"0a7ed36b-7186-452d-a777-a449b205ab66\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") "
Nov 02 14:46:52.700605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.700219    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7ed36b-7186-452d-a777-a449b205ab66-kube-api-access-67l9h\" (UniqueName: \"kubernetes.io/projected/0a7ed36b-7186-452d-a777-a449b205ab66-kube-api-access-67l9h\") pod \"0a7ed36b-7186-452d-a777-a449b205ab66\" (UID: \"0a7ed36b-7186-452d-a777-a449b205ab66\") "
Nov 02 14:46:52.700605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.700460    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-openshift-ca" pod "0a7ed36b-7186-452d-a777-a449b205ab66" (UID: "0a7ed36b-7186-452d-a777-a449b205ab66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:46:52.701540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.701511    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.702640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.702611    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-xks-ca" pod "0a7ed36b-7186-452d-a777-a449b205ab66" (UID: "0a7ed36b-7186-452d-a777-a449b205ab66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:46:52.703181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.703159    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7ed36b-7186-452d-a777-a449b205ab66-kube-api-access-67l9h" pod "0a7ed36b-7186-452d-a777-a449b205ab66" (UID: "0a7ed36b-7186-452d-a777-a449b205ab66"). InnerVolumeSpecName "kube-api-access-67l9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:46:52.803273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.803235    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67l9h\" (UniqueName: \"kubernetes.io/projected/0a7ed36b-7186-452d-a777-a449b205ab66-kube-api-access-67l9h\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:52.803273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:52.803272    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a7ed36b-7186-452d-a777-a449b205ab66-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:46:53.398866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:53.398822    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" path="/var/lib/kubelet/pods/2f1bf78b-d6af-48b8-b6c7-44c8f4694240/volumes"
Nov 02 14:46:53.411445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:53.411413    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n" event={"ID":"0a7ed36b-7186-452d-a777-a449b205ab66","Type":"ContainerDied","Data":"3ad7005cd3de0ec08b2be77f39f3a9ae0e352c42300abde47253d2c43428019b"}
Nov 02 14:46:53.411598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:53.411462    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ad7005cd3de0ec08b2be77f39f3a9ae0e352c42300abde47253d2c43428019b"
Nov 02 14:46:53.411598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:53.411517    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"
Nov 02 14:46:55.595871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:46:55.595837    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"]
Nov 02 14:47:00.596724 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.596674    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v" podUID="ca604bd3-98dc-408b-8c70-4ffc24eec28f" containerName="payload-processing" containerID="cri-o://c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c" gracePeriod=25
Nov 02 14:47:00.667121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.667095    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"]
Nov 02 14:47:00.667903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.667888    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0a7ed36b-7186-452d-a777-a449b205ab66" containerName="revoke-keys"
Nov 02 14:47:00.667963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.667904    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7ed36b-7186-452d-a777-a449b205ab66" containerName="revoke-keys"
Nov 02 14:47:00.667963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.667927    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy"
Nov 02 14:47:00.667963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.667933    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy"
Nov 02 14:47:00.667963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.667943    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy"
Nov 02 14:47:00.667963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.667949    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy"
Nov 02 14:47:00.668127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.668080    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy"
Nov 02 14:47:00.668127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.668101    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7ed36b-7186-452d-a777-a449b205ab66" containerName="revoke-keys"
Nov 02 14:47:00.672104 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.672086    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.695216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.695189    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"]
Nov 02 14:47:00.700591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.700095    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:47:00.700591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.700139    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:47:00.770156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.770107    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:47:00.770156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.770107    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:47:00.849862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.826362    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284z5\" (UniqueName: \"kubernetes.io/projected/86406096-39d6-494a-87ec-c720e81f9c8b-kube-api-access-284z5\") pod \"maas-api-revoke-sub-6a74376b-tprx4\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.849862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.826396    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a74376b-tprx4\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.849862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.826740    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a74376b-tprx4\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.915944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.915918    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:47:00.928451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.928425    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a74376b-tprx4\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.928611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.928574    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284z5\" (UniqueName: \"kubernetes.io/projected/86406096-39d6-494a-87ec-c720e81f9c8b-kube-api-access-284z5\") pod \"maas-api-revoke-sub-6a74376b-tprx4\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.928611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.928599    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a74376b-tprx4\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.929313 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.929291    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a74376b-tprx4\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.930821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.930768    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a74376b-tprx4\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.952785 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.952746    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284z5\" (UniqueName: \"kubernetes.io/projected/86406096-39d6-494a-87ec-c720e81f9c8b-kube-api-access-284z5\") pod \"maas-api-revoke-sub-6a74376b-tprx4\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:00.994835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:00.994806    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:01.030164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.030137    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca604bd3-98dc-408b-8c70-4ffc24eec28f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca604bd3-98dc-408b-8c70-4ffc24eec28f-plugins-config-volume\") pod \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\" (UID: \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\") "
Nov 02 14:47:01.030297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.030190    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca604bd3-98dc-408b-8c70-4ffc24eec28f-kube-api-access-qwnjf\" (UniqueName: \"kubernetes.io/projected/ca604bd3-98dc-408b-8c70-4ffc24eec28f-kube-api-access-qwnjf\") pod \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\" (UID: \"ca604bd3-98dc-408b-8c70-4ffc24eec28f\") "
Nov 02 14:47:01.030642 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.030606    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca604bd3-98dc-408b-8c70-4ffc24eec28f-plugins-config-volume" pod "ca604bd3-98dc-408b-8c70-4ffc24eec28f" (UID: "ca604bd3-98dc-408b-8c70-4ffc24eec28f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:01.032353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.032329    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca604bd3-98dc-408b-8c70-4ffc24eec28f-kube-api-access-qwnjf" pod "ca604bd3-98dc-408b-8c70-4ffc24eec28f" (UID: "ca604bd3-98dc-408b-8c70-4ffc24eec28f"). InnerVolumeSpecName "kube-api-access-qwnjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:01.132296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.132013    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca604bd3-98dc-408b-8c70-4ffc24eec28f-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:01.132611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.132583    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwnjf\" (UniqueName: \"kubernetes.io/projected/ca604bd3-98dc-408b-8c70-4ffc24eec28f-kube-api-access-qwnjf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:01.310355 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:47:01.310314    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86406096_39d6_494a_87ec_c720e81f9c8b.slice/crio-d8abc4d979fe87c0b5eb4c12fce81114f9608b9e4478dd52b73f46543857b934 WatchSource:0}: Error finding container d8abc4d979fe87c0b5eb4c12fce81114f9608b9e4478dd52b73f46543857b934: Status 404 returned error can't find the container with id d8abc4d979fe87c0b5eb4c12fce81114f9608b9e4478dd52b73f46543857b934
Nov 02 14:47:01.315515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.315479    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"]
Nov 02 14:47:01.494322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.494291    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4" event={"ID":"86406096-39d6-494a-87ec-c720e81f9c8b","Type":"ContainerStarted","Data":"d8abc4d979fe87c0b5eb4c12fce81114f9608b9e4478dd52b73f46543857b934"}
Nov 02 14:47:01.495926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.495896    5034 generic.go:356] "Generic (PLEG): container finished" podID="ca604bd3-98dc-408b-8c70-4ffc24eec28f" containerID="c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c" exitCode=0
Nov 02 14:47:01.496036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.495940    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"
Nov 02 14:47:01.496036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.495986    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v" event={"ID":"ca604bd3-98dc-408b-8c70-4ffc24eec28f","Type":"ContainerDied","Data":"c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c"}
Nov 02 14:47:01.496127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.496042    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v" event={"ID":"ca604bd3-98dc-408b-8c70-4ffc24eec28f","Type":"ContainerDied","Data":"995b14e3bff9ea171f205807fb03cfcf948f8611e60fb0a99f10796cf3efc285"}
Nov 02 14:47:01.496127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.496073    5034 scope.go:122] "RemoveContainer" containerID="c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c"
Nov 02 14:47:01.520696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.520664    5034 scope.go:122] "RemoveContainer" containerID="c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c"
Nov 02 14:47:01.521070 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:47:01.521035    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c\": container with ID starting with c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c not found: ID does not exist" containerID="c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c"
Nov 02 14:47:01.521140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.521082    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c"} err="failed to get container status \"c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c\": rpc error: code = NotFound desc = could not find container \"c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c\": container with ID starting with c0e0a62bce412e320af60e3f079f9b37fcb50e77eb126af44e986d66b0a4e97c not found: ID does not exist"
Nov 02 14:47:01.557207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.557178    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"]
Nov 02 14:47:01.577107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:01.577071    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cbde78-77994bf5d9-ppq6v"]
Nov 02 14:47:02.509396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:02.509355    5034 generic.go:356] "Generic (PLEG): container finished" podID="86406096-39d6-494a-87ec-c720e81f9c8b" containerID="34e4067ce969c9cfcc59491fdde6fc69a59039eb9d39cc327ba314c171acd1ca" exitCode=0
Nov 02 14:47:02.509814 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:02.509419    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4" event={"ID":"86406096-39d6-494a-87ec-c720e81f9c8b","Type":"ContainerDied","Data":"34e4067ce969c9cfcc59491fdde6fc69a59039eb9d39cc327ba314c171acd1ca"}
Nov 02 14:47:03.399427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.399394    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca604bd3-98dc-408b-8c70-4ffc24eec28f" path="/var/lib/kubelet/pods/ca604bd3-98dc-408b-8c70-4ffc24eec28f/volumes"
Nov 02 14:47:03.705937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.705913    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:03.875574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.875542    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-openshift-ca\") pod \"86406096-39d6-494a-87ec-c720e81f9c8b\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") "
Nov 02 14:47:03.875807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.875681    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-xks-ca\") pod \"86406096-39d6-494a-87ec-c720e81f9c8b\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") "
Nov 02 14:47:03.875807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.875726    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86406096-39d6-494a-87ec-c720e81f9c8b-kube-api-access-284z5\" (UniqueName: \"kubernetes.io/projected/86406096-39d6-494a-87ec-c720e81f9c8b-kube-api-access-284z5\") pod \"86406096-39d6-494a-87ec-c720e81f9c8b\" (UID: \"86406096-39d6-494a-87ec-c720e81f9c8b\") "
Nov 02 14:47:03.876014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.875984    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-openshift-ca" pod "86406096-39d6-494a-87ec-c720e81f9c8b" (UID: "86406096-39d6-494a-87ec-c720e81f9c8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:03.876439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.876413    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:03.877908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.877882    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-xks-ca" pod "86406096-39d6-494a-87ec-c720e81f9c8b" (UID: "86406096-39d6-494a-87ec-c720e81f9c8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:03.878381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.878355    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86406096-39d6-494a-87ec-c720e81f9c8b-kube-api-access-284z5" pod "86406096-39d6-494a-87ec-c720e81f9c8b" (UID: "86406096-39d6-494a-87ec-c720e81f9c8b"). InnerVolumeSpecName "kube-api-access-284z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:03.977953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.977879    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86406096-39d6-494a-87ec-c720e81f9c8b-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:03.977953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:03.977908    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284z5\" (UniqueName: \"kubernetes.io/projected/86406096-39d6-494a-87ec-c720e81f9c8b-kube-api-access-284z5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:04.528088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:04.528058    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"
Nov 02 14:47:04.528268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:04.528056    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4" event={"ID":"86406096-39d6-494a-87ec-c720e81f9c8b","Type":"ContainerDied","Data":"d8abc4d979fe87c0b5eb4c12fce81114f9608b9e4478dd52b73f46543857b934"}
Nov 02 14:47:04.528268 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:04.528181    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8abc4d979fe87c0b5eb4c12fce81114f9608b9e4478dd52b73f46543857b934"
Nov 02 14:47:18.421111 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:47:18.421014    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:47:37.559388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.559354    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"]
Nov 02 14:47:37.560206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.560191    5034 cpu_manager.go:414] "containerMap: removing container" podUID="86406096-39d6-494a-87ec-c720e81f9c8b" containerName="revoke-keys"
Nov 02 14:47:37.560267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.560207    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86406096-39d6-494a-87ec-c720e81f9c8b" containerName="revoke-keys"
Nov 02 14:47:37.560267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.560222    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ca604bd3-98dc-408b-8c70-4ffc24eec28f" containerName="payload-processing"
Nov 02 14:47:37.560267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.560228    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca604bd3-98dc-408b-8c70-4ffc24eec28f" containerName="payload-processing"
Nov 02 14:47:37.560384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.560372    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1bf78b-d6af-48b8-b6c7-44c8f4694240" containerName="istio-proxy"
Nov 02 14:47:37.560425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.560400    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="86406096-39d6-494a-87ec-c720e81f9c8b" containerName="revoke-keys"
Nov 02 14:47:37.560425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.560406    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca604bd3-98dc-408b-8c70-4ffc24eec28f" containerName="payload-processing"
Nov 02 14:47:37.564304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.564288    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.585977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.585949    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"]
Nov 02 14:47:37.594928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.594064    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:47:37.594928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.594144    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:47:37.594928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.594217    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:47:37.594928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.594807    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:47:37.645730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.645668    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20d2338-v4kz7\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.645964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.645806    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20d2338-v4kz7\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.646202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.646163    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6j4\" (UniqueName: \"kubernetes.io/projected/fa22c9ed-041e-4d67-af7b-4be66df234c5-kube-api-access-zl6j4\") pod \"maas-api-revoke-sub-d20d2338-v4kz7\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.747356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.747320    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6j4\" (UniqueName: \"kubernetes.io/projected/fa22c9ed-041e-4d67-af7b-4be66df234c5-kube-api-access-zl6j4\") pod \"maas-api-revoke-sub-d20d2338-v4kz7\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.747539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.747442    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20d2338-v4kz7\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.747539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.747482    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20d2338-v4kz7\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.748135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.748104    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20d2338-v4kz7\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.749838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.749813    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20d2338-v4kz7\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.775411 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.775382    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6j4\" (UniqueName: \"kubernetes.io/projected/fa22c9ed-041e-4d67-af7b-4be66df234c5-kube-api-access-zl6j4\") pod \"maas-api-revoke-sub-d20d2338-v4kz7\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:37.883024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:37.882926    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:38.210468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:38.210436    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"]
Nov 02 14:47:38.861253 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:38.861225    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7" event={"ID":"fa22c9ed-041e-4d67-af7b-4be66df234c5","Type":"ContainerStarted","Data":"58cef5ea54650dce0d3a7759745fe1a17f3bae4127b893970f5250b5a97f2338"}
Nov 02 14:47:38.861658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:38.861287    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7" event={"ID":"fa22c9ed-041e-4d67-af7b-4be66df234c5","Type":"ContainerStarted","Data":"b696d5985c75773653eff3f1bc86eeb5834581c0264e7f8205ef154ed41b2173"}
Nov 02 14:47:38.927844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:38.927739    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7" podStartSLOduration=1.927726443 podStartE2EDuration="1.927726443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:47:38.923427227 +0000 UTC m=+3849.674731668" watchObservedRunningTime="2026-11-02 14:47:38.927726443 +0000 UTC m=+3849.679030881"
Nov 02 14:47:39.871562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:39.871523    5034 generic.go:356] "Generic (PLEG): container finished" podID="fa22c9ed-041e-4d67-af7b-4be66df234c5" containerID="58cef5ea54650dce0d3a7759745fe1a17f3bae4127b893970f5250b5a97f2338" exitCode=0
Nov 02 14:47:39.872002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:39.871587    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7" event={"ID":"fa22c9ed-041e-4d67-af7b-4be66df234c5","Type":"ContainerDied","Data":"58cef5ea54650dce0d3a7759745fe1a17f3bae4127b893970f5250b5a97f2338"}
Nov 02 14:47:41.058542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.058513    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:41.210073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.209981    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa22c9ed-041e-4d67-af7b-4be66df234c5-kube-api-access-zl6j4\" (UniqueName: \"kubernetes.io/projected/fa22c9ed-041e-4d67-af7b-4be66df234c5-kube-api-access-zl6j4\") pod \"fa22c9ed-041e-4d67-af7b-4be66df234c5\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") "
Nov 02 14:47:41.210269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.210186    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-xks-ca\") pod \"fa22c9ed-041e-4d67-af7b-4be66df234c5\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") "
Nov 02 14:47:41.210350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.210330    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-openshift-ca\") pod \"fa22c9ed-041e-4d67-af7b-4be66df234c5\" (UID: \"fa22c9ed-041e-4d67-af7b-4be66df234c5\") "
Nov 02 14:47:41.210833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.210802    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-openshift-ca" pod "fa22c9ed-041e-4d67-af7b-4be66df234c5" (UID: "fa22c9ed-041e-4d67-af7b-4be66df234c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:41.212599 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.212569    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa22c9ed-041e-4d67-af7b-4be66df234c5-kube-api-access-zl6j4" pod "fa22c9ed-041e-4d67-af7b-4be66df234c5" (UID: "fa22c9ed-041e-4d67-af7b-4be66df234c5"). InnerVolumeSpecName "kube-api-access-zl6j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:41.213126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.213098    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-xks-ca" pod "fa22c9ed-041e-4d67-af7b-4be66df234c5" (UID: "fa22c9ed-041e-4d67-af7b-4be66df234c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:41.312481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.312444    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:41.312481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.312474    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa22c9ed-041e-4d67-af7b-4be66df234c5-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:41.312481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.312484    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl6j4\" (UniqueName: \"kubernetes.io/projected/fa22c9ed-041e-4d67-af7b-4be66df234c5-kube-api-access-zl6j4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:41.890444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.890403    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7" event={"ID":"fa22c9ed-041e-4d67-af7b-4be66df234c5","Type":"ContainerDied","Data":"b696d5985c75773653eff3f1bc86eeb5834581c0264e7f8205ef154ed41b2173"}
Nov 02 14:47:41.890444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.890449    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"
Nov 02 14:47:41.890716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:41.890464    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b696d5985c75773653eff3f1bc86eeb5834581c0264e7f8205ef154ed41b2173"
Nov 02 14:47:48.415212 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:47:48.415172    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:47:54.714704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.714668    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"]
Nov 02 14:47:54.715845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.715820    5034 cpu_manager.go:414] "containerMap: removing container" podUID="fa22c9ed-041e-4d67-af7b-4be66df234c5" containerName="revoke-keys"
Nov 02 14:47:54.715845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.715844    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa22c9ed-041e-4d67-af7b-4be66df234c5" containerName="revoke-keys"
Nov 02 14:47:54.716088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.716073    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa22c9ed-041e-4d67-af7b-4be66df234c5" containerName="revoke-keys"
Nov 02 14:47:54.720469 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.720450    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:54.738112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.738085    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"]
Nov 02 14:47:54.746396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.746349    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:47:54.746396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.746369    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:47:54.746583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.746440    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:47:54.750675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.750654    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:47:54.781736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.781711    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95607abd-8ng7r\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:54.781848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.781758    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlhpl\" (UniqueName: \"kubernetes.io/projected/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-kube-api-access-rlhpl\") pod \"maas-api-revoke-sub-95607abd-8ng7r\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:54.781908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.781878    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95607abd-8ng7r\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:54.883456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.883431    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95607abd-8ng7r\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:54.883583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.883474    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlhpl\" (UniqueName: \"kubernetes.io/projected/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-kube-api-access-rlhpl\") pod \"maas-api-revoke-sub-95607abd-8ng7r\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:54.883583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.883537    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95607abd-8ng7r\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:54.884265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.884239    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95607abd-8ng7r\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:54.885839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.885818    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95607abd-8ng7r\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:54.908879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:54.908855    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlhpl\" (UniqueName: \"kubernetes.io/projected/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-kube-api-access-rlhpl\") pod \"maas-api-revoke-sub-95607abd-8ng7r\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:55.035662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.035552    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:55.380390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.376671    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"]
Nov 02 14:47:55.408749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.407402    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"]
Nov 02 14:47:55.414740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.414718    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.434875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.434848    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"]
Nov 02 14:47:55.553371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.553343    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.553547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.553404    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.553547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.553472    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.553547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.553537    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdca348-be51-47a6-bf14-32e7c181f160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.553666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.553575    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdca348-be51-47a6-bf14-32e7c181f160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.553666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.553623    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmlp4\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-kube-api-access-dmlp4\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.553737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.553697    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.553812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.553759    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.553930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.553912    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655145    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdca348-be51-47a6-bf14-32e7c181f160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655184    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdca348-be51-47a6-bf14-32e7c181f160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655223    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmlp4\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-kube-api-access-dmlp4\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655332    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655357    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655496    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655576    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655650    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655692    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655739    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.655861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655768    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.656003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.655984    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdca348-be51-47a6-bf14-32e7c181f160-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.656057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.656025    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.656100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.656067    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.657745 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.657724    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.657956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.657936    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdca348-be51-47a6-bf14-32e7c181f160-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.680133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.680104    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmlp4\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-kube-api-access-dmlp4\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.680233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.680108    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-l2jwp\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:55.768559 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:55.768536    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:56.019841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:56.019771    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r" event={"ID":"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61","Type":"ContainerStarted","Data":"65a26b7e13ca08ad1ba4cc9f8a9f6f20f64f1cf782da31991e580e1184b46b9c"}
Nov 02 14:47:56.020047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:56.019859    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r" event={"ID":"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61","Type":"ContainerStarted","Data":"c6c9f6051fbf42e55d63e62de147b909845d2e76b080e1260c5cb9d460ca427e"}
Nov 02 14:47:56.082877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:56.082829    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r" podStartSLOduration=2.082815502 podStartE2EDuration="2.082815502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:47:56.077153056 +0000 UTC m=+3866.828457504" watchObservedRunningTime="2026-11-02 14:47:56.082815502 +0000 UTC m=+3866.834119943"
Nov 02 14:47:56.099720 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:47:56.099692    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdca348_be51_47a6_bf14_32e7c181f160.slice/crio-3108ab13ef19075e9710ac25f064983723a2c06af1d773c7b54acf8f1aa01386 WatchSource:0}: Error finding container 3108ab13ef19075e9710ac25f064983723a2c06af1d773c7b54acf8f1aa01386: Status 404 returned error can't find the container with id 3108ab13ef19075e9710ac25f064983723a2c06af1d773c7b54acf8f1aa01386
Nov 02 14:47:56.102340 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:56.102312    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"]
Nov 02 14:47:56.103656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:56.103616    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:47:56.103755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:56.103705    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:47:56.103755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:56.103746    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:47:57.031519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:57.031484    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" event={"ID":"0bdca348-be51-47a6-bf14-32e7c181f160","Type":"ContainerStarted","Data":"3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166"}
Nov 02 14:47:57.032021 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:57.031541    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" event={"ID":"0bdca348-be51-47a6-bf14-32e7c181f160","Type":"ContainerStarted","Data":"3108ab13ef19075e9710ac25f064983723a2c06af1d773c7b54acf8f1aa01386"}
Nov 02 14:47:57.033352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:57.033327    5034 generic.go:356] "Generic (PLEG): container finished" podID="98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61" containerID="65a26b7e13ca08ad1ba4cc9f8a9f6f20f64f1cf782da31991e580e1184b46b9c" exitCode=0
Nov 02 14:47:57.033467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:57.033388    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r" event={"ID":"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61","Type":"ContainerDied","Data":"65a26b7e13ca08ad1ba4cc9f8a9f6f20f64f1cf782da31991e580e1184b46b9c"}
Nov 02 14:47:57.112638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:57.112586    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" podStartSLOduration=2.112572143 podStartE2EDuration="2.112572143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:47:57.107442495 +0000 UTC m=+3867.858746944" watchObservedRunningTime="2026-11-02 14:47:57.112572143 +0000 UTC m=+3867.863876578"
Nov 02 14:47:57.780802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:57.773092    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:47:57.784797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:57.783895    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l2jwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body=
Nov 02 14:47:57.784797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:57.783974    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused"
Nov 02 14:47:58.264198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.264171    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:58.411589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.411501    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-xks-ca\") pod \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") "
Nov 02 14:47:58.411589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.411578    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-kube-api-access-rlhpl\" (UniqueName: \"kubernetes.io/projected/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-kube-api-access-rlhpl\") pod \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") "
Nov 02 14:47:58.411875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.411646    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-openshift-ca\") pod \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\" (UID: \"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61\") "
Nov 02 14:47:58.412042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.412009    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-openshift-ca" pod "98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61" (UID: "98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:47:58.412365 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.412343    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:58.413940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.413909    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-kube-api-access-rlhpl" pod "98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61" (UID: "98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61"). InnerVolumeSpecName "kube-api-access-rlhpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:47:58.414207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.414189    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-xks-ca" pod "98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61" (UID: "98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:47:58.513457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.513422    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:58.513457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.513453    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlhpl\" (UniqueName: \"kubernetes.io/projected/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61-kube-api-access-rlhpl\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:47:58.779803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.773137    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l2jwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body=
Nov 02 14:47:58.779803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:58.773197    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused"
Nov 02 14:47:59.051274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:59.051199    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"
Nov 02 14:47:59.051274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:59.051227    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r" event={"ID":"98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61","Type":"ContainerDied","Data":"c6c9f6051fbf42e55d63e62de147b909845d2e76b080e1260c5cb9d460ca427e"}
Nov 02 14:47:59.051470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:59.051281    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6c9f6051fbf42e55d63e62de147b909845d2e76b080e1260c5cb9d460ca427e"
Nov 02 14:47:59.790922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:59.790880    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l2jwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body=
Nov 02 14:47:59.791537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:47:59.791501    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused"
Nov 02 14:48:00.777805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:00.770159    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l2jwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body=
Nov 02 14:48:00.777805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:00.770219    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused"
Nov 02 14:48:01.786846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:01.777126    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l2jwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body=
Nov 02 14:48:01.786846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:01.777186    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused"
Nov 02 14:48:02.773216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:02.773176    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l2jwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body=
Nov 02 14:48:02.773474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:02.773445    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused"
Nov 02 14:48:03.769548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:03.769513    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-l2jwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused" start-of-body=
Nov 02 14:48:03.770100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:03.769566    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.146:15021/healthz/ready\": dial tcp 10.132.1.146:15021: connect: connection refused"
Nov 02 14:48:05.291210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:05.291172    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:48:05.291727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:05.291654    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:48:05.292549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:05.292521    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:48:06.276828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.276790    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"]
Nov 02 14:48:06.278179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.278152    5034 cpu_manager.go:414] "containerMap: removing container" podUID="98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61" containerName="revoke-keys"
Nov 02 14:48:06.278179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.278177    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61" containerName="revoke-keys"
Nov 02 14:48:06.279362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.278472    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61" containerName="revoke-keys"
Nov 02 14:48:06.283511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.283490    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.312618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.306514    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:48:06.312618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.306877    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:48:06.312618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.310286    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:48:06.312618 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.311211    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"]
Nov 02 14:48:06.317798 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.313400    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:48:06.368247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.368216    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ntz\" (UniqueName: \"kubernetes.io/projected/a2bf2265-7b80-4403-8844-71b555a59bcc-kube-api-access-68ntz\") pod \"maas-api-revoke-sub-691c3c9a-9j27q\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.368406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.368351    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-691c3c9a-9j27q\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.368471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.368460    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-691c3c9a-9j27q\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.469740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.469705    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-691c3c9a-9j27q\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.469946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.469885    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ntz\" (UniqueName: \"kubernetes.io/projected/a2bf2265-7b80-4403-8844-71b555a59bcc-kube-api-access-68ntz\") pod \"maas-api-revoke-sub-691c3c9a-9j27q\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.470014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.469963    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-691c3c9a-9j27q\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.470732 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.470704    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-691c3c9a-9j27q\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.472246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.472212    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-691c3c9a-9j27q\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.495693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.495670    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ntz\" (UniqueName: \"kubernetes.io/projected/a2bf2265-7b80-4403-8844-71b555a59bcc-kube-api-access-68ntz\") pod \"maas-api-revoke-sub-691c3c9a-9j27q\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:06.612647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:06.612580    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:07.126921 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:48:07.126885    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bf2265_7b80_4403_8844_71b555a59bcc.slice/crio-9828b3cadae38e2829006266ab8f0a958ccdaa5c09c97fdd993cb2c7a5f85e44 WatchSource:0}: Error finding container 9828b3cadae38e2829006266ab8f0a958ccdaa5c09c97fdd993cb2c7a5f85e44: Status 404 returned error can't find the container with id 9828b3cadae38e2829006266ab8f0a958ccdaa5c09c97fdd993cb2c7a5f85e44
Nov 02 14:48:07.129263 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:07.129238    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"]
Nov 02 14:48:07.162153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:07.162121    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q" event={"ID":"a2bf2265-7b80-4403-8844-71b555a59bcc","Type":"ContainerStarted","Data":"9828b3cadae38e2829006266ab8f0a958ccdaa5c09c97fdd993cb2c7a5f85e44"}
Nov 02 14:48:08.175330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:08.175291    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q" event={"ID":"a2bf2265-7b80-4403-8844-71b555a59bcc","Type":"ContainerStarted","Data":"ed864a37e84c60a7c1b5ce887f7b976c6cccfa3037a6ce3e615c3a0bf648d1fe"}
Nov 02 14:48:08.226155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:08.226084    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q" podStartSLOduration=2.226063101 podStartE2EDuration="2.226063101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:48:08.218088654 +0000 UTC m=+3878.969393096" watchObservedRunningTime="2026-11-02 14:48:08.226063101 +0000 UTC m=+3878.977367540"
Nov 02 14:48:09.191519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:09.191477    5034 generic.go:356] "Generic (PLEG): container finished" podID="a2bf2265-7b80-4403-8844-71b555a59bcc" containerID="ed864a37e84c60a7c1b5ce887f7b976c6cccfa3037a6ce3e615c3a0bf648d1fe" exitCode=0
Nov 02 14:48:09.192182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:09.191571    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q" event={"ID":"a2bf2265-7b80-4403-8844-71b555a59bcc","Type":"ContainerDied","Data":"ed864a37e84c60a7c1b5ce887f7b976c6cccfa3037a6ce3e615c3a0bf648d1fe"}
Nov 02 14:48:10.413282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.413254    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:10.455015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.454918    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"]
Nov 02 14:48:10.457245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.457214    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a2bf2265-7b80-4403-8844-71b555a59bcc" containerName="revoke-keys"
Nov 02 14:48:10.457425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.457410    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2bf2265-7b80-4403-8844-71b555a59bcc" containerName="revoke-keys"
Nov 02 14:48:10.457949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.457927    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2bf2265-7b80-4403-8844-71b555a59bcc" containerName="revoke-keys"
Nov 02 14:48:10.465590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.465559    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-openshift-ca\") pod \"a2bf2265-7b80-4403-8844-71b555a59bcc\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") "
Nov 02 14:48:10.465893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.465872    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-xks-ca\") pod \"a2bf2265-7b80-4403-8844-71b555a59bcc\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") "
Nov 02 14:48:10.466051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.466034    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2bf2265-7b80-4403-8844-71b555a59bcc-kube-api-access-68ntz\" (UniqueName: \"kubernetes.io/projected/a2bf2265-7b80-4403-8844-71b555a59bcc-kube-api-access-68ntz\") pod \"a2bf2265-7b80-4403-8844-71b555a59bcc\" (UID: \"a2bf2265-7b80-4403-8844-71b555a59bcc\") "
Nov 02 14:48:10.470262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.470233    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-openshift-ca" pod "a2bf2265-7b80-4403-8844-71b555a59bcc" (UID: "a2bf2265-7b80-4403-8844-71b555a59bcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:10.478208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.478176    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-xks-ca" pod "a2bf2265-7b80-4403-8844-71b555a59bcc" (UID: "a2bf2265-7b80-4403-8844-71b555a59bcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:48:10.485654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.485190    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.486478 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.486437    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"]
Nov 02 14:48:10.542225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.540988    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bf2265-7b80-4403-8844-71b555a59bcc-kube-api-access-68ntz" pod "a2bf2265-7b80-4403-8844-71b555a59bcc" (UID: "a2bf2265-7b80-4403-8844-71b555a59bcc"). InnerVolumeSpecName "kube-api-access-68ntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:10.568302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.568264    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.568472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.568422    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.568472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.568464    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.568625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.568606    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.568701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.568660    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1666e4c-a00a-408b-8fd5-4570510a96c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.568857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.568818    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.568990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.568917    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwv4\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-kube-api-access-nwwv4\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.568990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.568954    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.568990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.568987    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.569194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.569064    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:10.569194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.569076    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2bf2265-7b80-4403-8844-71b555a59bcc-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:10.569194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.569084    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ntz\" (UniqueName: \"kubernetes.io/projected/a2bf2265-7b80-4403-8844-71b555a59bcc-kube-api-access-68ntz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:10.669958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.669927    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.669978    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1666e4c-a00a-408b-8fd5-4570510a96c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670060    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670152    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwv4\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-kube-api-access-nwwv4\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670188    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670233    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670288    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670371    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670386    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670423    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670467    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670802    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.670824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.670809    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1666e4c-a00a-408b-8fd5-4570510a96c4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.671246 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.671218    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.672653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.672624    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.673042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.673022    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.695251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.695225    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.695463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.695447    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwv4\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-kube-api-access-nwwv4\") pod \"maas-default-gateway-openshift-default-6886b96f9-n62n9\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:10.836959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:10.836931    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:11.163268 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:48:11.163226    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1666e4c_a00a_408b_8fd5_4570510a96c4.slice/crio-578f0f9ce46dd50c43df6c846965b713925e51c004a1f8a82d1cd5f59424b65c WatchSource:0}: Error finding container 578f0f9ce46dd50c43df6c846965b713925e51c004a1f8a82d1cd5f59424b65c: Status 404 returned error can't find the container with id 578f0f9ce46dd50c43df6c846965b713925e51c004a1f8a82d1cd5f59424b65c
Nov 02 14:48:11.166014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:11.165986    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"]
Nov 02 14:48:11.166419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:11.166387    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:48:11.166513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:11.166471    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:48:11.166585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:11.166517    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:48:11.215218 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:11.215191    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q" event={"ID":"a2bf2265-7b80-4403-8844-71b555a59bcc","Type":"ContainerDied","Data":"9828b3cadae38e2829006266ab8f0a958ccdaa5c09c97fdd993cb2c7a5f85e44"}
Nov 02 14:48:11.215355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:11.215246    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9828b3cadae38e2829006266ab8f0a958ccdaa5c09c97fdd993cb2c7a5f85e44"
Nov 02 14:48:11.215355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:11.215343    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"
Nov 02 14:48:11.217140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:11.217101    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" event={"ID":"e1666e4c-a00a-408b-8fd5-4570510a96c4","Type":"ContainerStarted","Data":"578f0f9ce46dd50c43df6c846965b713925e51c004a1f8a82d1cd5f59424b65c"}
Nov 02 14:48:12.228165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:12.228129    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" event={"ID":"e1666e4c-a00a-408b-8fd5-4570510a96c4","Type":"ContainerStarted","Data":"8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6"}
Nov 02 14:48:12.296801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:12.296367    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" podStartSLOduration=2.296345585 podStartE2EDuration="2.296345585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:48:12.281247862 +0000 UTC m=+3883.032552302" watchObservedRunningTime="2026-11-02 14:48:12.296345585 +0000 UTC m=+3883.047650024"
Nov 02 14:48:12.837087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:12.837052    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:12.838978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:12.838949    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-n62n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body=
Nov 02 14:48:12.839067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:12.838997    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused"
Nov 02 14:48:13.837875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:13.837844    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-n62n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body=
Nov 02 14:48:13.838299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:13.837919    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused"
Nov 02 14:48:14.838010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:14.837973    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-n62n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body=
Nov 02 14:48:14.838526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:14.838149    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused"
Nov 02 14:48:15.837650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:15.837612    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-n62n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body=
Nov 02 14:48:15.837858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:15.837667    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused"
Nov 02 14:48:16.837292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:16.837255    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-n62n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body=
Nov 02 14:48:16.837671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:16.837314    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused"
Nov 02 14:48:17.837594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:17.837557    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-n62n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body=
Nov 02 14:48:17.838136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:17.837604    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused"
Nov 02 14:48:18.421376 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:48:18.421335    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:48:18.838173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:18.838137    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-n62n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused" start-of-body=
Nov 02 14:48:18.838598 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:18.838199    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.148:15021/healthz/ready\": dial tcp 10.132.1.148:15021: connect: connection refused"
Nov 02 14:48:19.860947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:19.860917    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:19.862965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:19.862218    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:19.863391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:19.863368    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:48:30.913078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:30.913043    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:48:30.922601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:30.922570    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:48:31.402499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:31.402463    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"]
Nov 02 14:48:36.403653 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:36.403611    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg" podUID="02d5f06f-cab1-4546-a546-95fc619bce7b" containerName="payload-processing" containerID="cri-o://0f5b02fc3d4b8a0fd561addf7c17321528ad0349a81b74ec4e266cef703388bc" gracePeriod=25
Nov 02 14:48:39.196905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:39.196863    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg" podUID="02d5f06f-cab1-4546-a546-95fc619bce7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:48:48.032429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:48.032354    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"]
Nov 02 14:48:48.048589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:48.048557    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d99bc3-h2skq"]
Nov 02 14:48:48.418293 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:48:48.418214    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:48:49.034393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:49.034358    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"]
Nov 02 14:48:49.050099 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:49.050066    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f95719f-m8v4q"]
Nov 02 14:48:49.197037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:49.196996    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg" podUID="02d5f06f-cab1-4546-a546-95fc619bce7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:48:49.398468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:49.398395    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0524cef3-0ff9-49c1-b317-4cfd9d868a57" path="/var/lib/kubelet/pods/0524cef3-0ff9-49c1-b317-4cfd9d868a57/volumes"
Nov 02 14:48:49.398930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:49.398914    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af17e386-2606-46b4-896a-4d4ddaa3656b" path="/var/lib/kubelet/pods/af17e386-2606-46b4-896a-4d4ddaa3656b/volumes"
Nov 02 14:48:52.209065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:52.209030    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"]
Nov 02 14:48:52.209884 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:52.209852    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy" containerID="cri-o://f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9" gracePeriod=30
Nov 02 14:48:53.884526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.884491    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"]
Nov 02 14:48:53.892026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.892002    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:53.913116 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.912648    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"]
Nov 02 14:48:53.922195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.920445    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e16d1c0e-openshift-default-dockercfg-6mjr7\""
Nov 02 14:48:53.976156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.976121    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-podinfo\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:53.976325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.976197    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c7637-7247-4cf6-a551-02f2fc51c21c-istiod-ca-cert\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:53.976325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.976275    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-data\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:53.976459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.976334    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tn5b\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-kube-api-access-2tn5b\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:53.976459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.976418    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-token\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:53.976578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.976521    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-socket\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:53.976636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.976622    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-envoy\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:53.976711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.976694    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-credential-socket\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:53.976838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:53.976814    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-certs\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.079801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.079739    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-podinfo\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.080097 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.080077    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c7637-7247-4cf6-a551-02f2fc51c21c-istiod-ca-cert\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.080248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.080233    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-data\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.080388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.080375    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tn5b\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-kube-api-access-2tn5b\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.080502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.080487    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-token\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.080802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.080768    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-socket\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.080969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.080955    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-envoy\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.081126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.081112    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-credential-socket\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.081298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.081286    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-certs\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.081945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.081922    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-certs\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.095726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.089656    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-data\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.095726 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.090349    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c7637-7247-4cf6-a551-02f2fc51c21c-istiod-ca-cert\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.096909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.096569    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-credential-socket\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.096909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.096851    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-socket\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.098165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.097905    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-podinfo\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.099633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.098989    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-envoy\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.140867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.140772    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tn5b\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-kube-api-access-2tn5b\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.146746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.142189    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-token\") pod \"e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") " pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.230018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.229978    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:54.575603 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:48:54.575567    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2c7637_7247_4cf6_a551_02f2fc51c21c.slice/crio-71a6a826f16e4cd03c55f37f5ddc2936b6f658ef8dca57008553775e37281f19 WatchSource:0}: Error finding container 71a6a826f16e4cd03c55f37f5ddc2936b6f658ef8dca57008553775e37281f19: Status 404 returned error can't find the container with id 71a6a826f16e4cd03c55f37f5ddc2936b6f658ef8dca57008553775e37281f19
Nov 02 14:48:54.577553 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.577490    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"]
Nov 02 14:48:54.578880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.578847    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:48:54.579001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.578925    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:48:54.579001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.578971    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:48:54.666591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:54.666557    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" event={"ID":"be2c7637-7247-4cf6-a551-02f2fc51c21c","Type":"ContainerStarted","Data":"71a6a826f16e4cd03c55f37f5ddc2936b6f658ef8dca57008553775e37281f19"}
Nov 02 14:48:55.678060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:55.678024    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" event={"ID":"be2c7637-7247-4cf6-a551-02f2fc51c21c","Type":"ContainerStarted","Data":"6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c"}
Nov 02 14:48:55.742458 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:55.742398    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" podStartSLOduration=2.742379883 podStartE2EDuration="2.742379883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:48:55.739676156 +0000 UTC m=+3926.490980593" watchObservedRunningTime="2026-11-02 14:48:55.742379883 +0000 UTC m=+3926.493684322"
Nov 02 14:48:56.231286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.231253    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:56.237491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.237467    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:56.689876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.689840    5034 generic.go:356] "Generic (PLEG): container finished" podID="02d5f06f-cab1-4546-a546-95fc619bce7b" containerID="0f5b02fc3d4b8a0fd561addf7c17321528ad0349a81b74ec4e266cef703388bc" exitCode=0
Nov 02 14:48:56.690336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.689910    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg" event={"ID":"02d5f06f-cab1-4546-a546-95fc619bce7b","Type":"ContainerDied","Data":"0f5b02fc3d4b8a0fd561addf7c17321528ad0349a81b74ec4e266cef703388bc"}
Nov 02 14:48:56.690336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.689964    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg" event={"ID":"02d5f06f-cab1-4546-a546-95fc619bce7b","Type":"ContainerDied","Data":"d47234ddcb166ad61f59a2f6c1e8ad6a567e755202587c52f5a09a05a77bd723"}
Nov 02 14:48:56.690336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.689979    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d47234ddcb166ad61f59a2f6c1e8ad6a567e755202587c52f5a09a05a77bd723"
Nov 02 14:48:56.690508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.690461    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:56.691475 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.691450    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:48:56.720762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.720739    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:48:56.840038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.839957    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d5f06f-cab1-4546-a546-95fc619bce7b-kube-api-access-tpkn8\" (UniqueName: \"kubernetes.io/projected/02d5f06f-cab1-4546-a546-95fc619bce7b-kube-api-access-tpkn8\") pod \"02d5f06f-cab1-4546-a546-95fc619bce7b\" (UID: \"02d5f06f-cab1-4546-a546-95fc619bce7b\") "
Nov 02 14:48:56.840243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.840132    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d5f06f-cab1-4546-a546-95fc619bce7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d5f06f-cab1-4546-a546-95fc619bce7b-plugins-config-volume\") pod \"02d5f06f-cab1-4546-a546-95fc619bce7b\" (UID: \"02d5f06f-cab1-4546-a546-95fc619bce7b\") "
Nov 02 14:48:56.840586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.840554    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d5f06f-cab1-4546-a546-95fc619bce7b-plugins-config-volume" pod "02d5f06f-cab1-4546-a546-95fc619bce7b" (UID: "02d5f06f-cab1-4546-a546-95fc619bce7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:56.841228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.841204    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02d5f06f-cab1-4546-a546-95fc619bce7b-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:56.858423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.842386    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d5f06f-cab1-4546-a546-95fc619bce7b-kube-api-access-tpkn8" pod "02d5f06f-cab1-4546-a546-95fc619bce7b" (UID: "02d5f06f-cab1-4546-a546-95fc619bce7b"). InnerVolumeSpecName "kube-api-access-tpkn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:56.942344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:56.942308    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpkn8\" (UniqueName: \"kubernetes.io/projected/02d5f06f-cab1-4546-a546-95fc619bce7b-kube-api-access-tpkn8\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.521409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.521379    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:48:57.652303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652273    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-istio-token\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-istio-token\") pod \"d157582e-3896-46f9-a053-010c45191ec3\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") "
Nov 02 14:48:57.652465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652312    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-credential-socket\") pod \"d157582e-3896-46f9-a053-010c45191ec3\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") "
Nov 02 14:48:57.652465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652334    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-data\") pod \"d157582e-3896-46f9-a053-010c45191ec3\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") "
Nov 02 14:48:57.652465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652367    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-socket\") pod \"d157582e-3896-46f9-a053-010c45191ec3\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") "
Nov 02 14:48:57.652661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652470    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-certs\") pod \"d157582e-3896-46f9-a053-010c45191ec3\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") "
Nov 02 14:48:57.652661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652592    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-credential-socket" pod "d157582e-3896-46f9-a053-010c45191ec3" (UID: "d157582e-3896-46f9-a053-010c45191ec3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:48:57.652661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652615    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-kube-api-access-k8pwm\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-kube-api-access-k8pwm\") pod \"d157582e-3896-46f9-a053-010c45191ec3\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") "
Nov 02 14:48:57.652911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652714    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-data" pod "d157582e-3896-46f9-a053-010c45191ec3" (UID: "d157582e-3896-46f9-a053-010c45191ec3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:48:57.652911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652726    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-socket" pod "d157582e-3896-46f9-a053-010c45191ec3" (UID: "d157582e-3896-46f9-a053-010c45191ec3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:48:57.652911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652721    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-envoy\") pod \"d157582e-3896-46f9-a053-010c45191ec3\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") "
Nov 02 14:48:57.653112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652967    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-certs" pod "d157582e-3896-46f9-a053-010c45191ec3" (UID: "d157582e-3896-46f9-a053-010c45191ec3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:48:57.653112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.652992    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d157582e-3896-46f9-a053-010c45191ec3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d157582e-3896-46f9-a053-010c45191ec3-istio-podinfo\") pod \"d157582e-3896-46f9-a053-010c45191ec3\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") "
Nov 02 14:48:57.653112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.653035    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d157582e-3896-46f9-a053-010c45191ec3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d157582e-3896-46f9-a053-010c45191ec3-istiod-ca-cert\") pod \"d157582e-3896-46f9-a053-010c45191ec3\" (UID: \"d157582e-3896-46f9-a053-010c45191ec3\") "
Nov 02 14:48:57.653746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.653719    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d157582e-3896-46f9-a053-010c45191ec3-istiod-ca-cert" pod "d157582e-3896-46f9-a053-010c45191ec3" (UID: "d157582e-3896-46f9-a053-010c45191ec3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:48:57.653978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.653955    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d157582e-3896-46f9-a053-010c45191ec3-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.654066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.653984    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.654066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.653999    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.654066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.654014    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.654066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.654030    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.654988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.654943    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-istio-token" pod "d157582e-3896-46f9-a053-010c45191ec3" (UID: "d157582e-3896-46f9-a053-010c45191ec3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:57.655207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.655187    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-kube-api-access-k8pwm" pod "d157582e-3896-46f9-a053-010c45191ec3" (UID: "d157582e-3896-46f9-a053-010c45191ec3"). InnerVolumeSpecName "kube-api-access-k8pwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:48:57.655314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.655296    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-envoy" pod "d157582e-3896-46f9-a053-010c45191ec3" (UID: "d157582e-3896-46f9-a053-010c45191ec3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:48:57.655845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.655823    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d157582e-3896-46f9-a053-010c45191ec3-istio-podinfo" pod "d157582e-3896-46f9-a053-010c45191ec3" (UID: "d157582e-3896-46f9-a053-010c45191ec3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:48:57.703746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.703715    5034 generic.go:356] "Generic (PLEG): container finished" podID="d157582e-3896-46f9-a053-010c45191ec3" containerID="f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9" exitCode=0
Nov 02 14:48:57.703746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.703744    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"
Nov 02 14:48:57.704213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.703801    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" event={"ID":"d157582e-3896-46f9-a053-010c45191ec3","Type":"ContainerDied","Data":"f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9"}
Nov 02 14:48:57.704213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.703865    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx" event={"ID":"d157582e-3896-46f9-a053-010c45191ec3","Type":"ContainerDied","Data":"81bd2045bf580163ca6f019c5502e7e370b477eba7a92a0a948b9e11f8c0de23"}
Nov 02 14:48:57.704213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.703887    5034 scope.go:122] "RemoveContainer" containerID="f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9"
Nov 02 14:48:57.704213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.703947    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"
Nov 02 14:48:57.718526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.718507    5034 scope.go:122] "RemoveContainer" containerID="f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9"
Nov 02 14:48:57.719856 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:48:57.719828    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9\": container with ID starting with f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9 not found: ID does not exist" containerID="f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9"
Nov 02 14:48:57.720011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.719867    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9"} err="failed to get container status \"f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9\": rpc error: code = NotFound desc = could not find container \"f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9\": container with ID starting with f2e56dcd83334fae90025f33d44a261d265263b692a87ef08373166201370de9 not found: ID does not exist"
Nov 02 14:48:57.755826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.755757    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d157582e-3896-46f9-a053-010c45191ec3-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.755826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.755798    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.755826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.755812    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8pwm\" (UniqueName: \"kubernetes.io/projected/d157582e-3896-46f9-a053-010c45191ec3-kube-api-access-k8pwm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.755826 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.755824    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d157582e-3896-46f9-a053-010c45191ec3-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:48:57.763467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.763447    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"]
Nov 02 14:48:57.767854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.767835    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c8f647-7586b4d465-5prgg"]
Nov 02 14:48:57.804146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.804117    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"]
Nov 02 14:48:57.810255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:57.810232    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ea3e8f-openshift-default-7b945d6cc7-psjhx"]
Nov 02 14:48:59.033024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:59.032989    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"]
Nov 02 14:48:59.046674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:59.046643    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c5d9c1c-2pv99"]
Nov 02 14:48:59.398133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:59.398064    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d5f06f-cab1-4546-a546-95fc619bce7b" path="/var/lib/kubelet/pods/02d5f06f-cab1-4546-a546-95fc619bce7b/volumes"
Nov 02 14:48:59.398518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:59.398504    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21010870-cadd-4750-a4b4-c1be487a6424" path="/var/lib/kubelet/pods/21010870-cadd-4750-a4b4-c1be487a6424/volumes"
Nov 02 14:48:59.398978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:48:59.398964    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d157582e-3896-46f9-a053-010c45191ec3" path="/var/lib/kubelet/pods/d157582e-3896-46f9-a053-010c45191ec3/volumes"
Nov 02 14:49:18.415592 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:49:18.415553    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:49:21.754312 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.754277    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"]
Nov 02 14:49:21.755095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.755080    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy"
Nov 02 14:49:21.755095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.755096    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy"
Nov 02 14:49:21.755186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.755142    5034 cpu_manager.go:414] "containerMap: removing container" podUID="02d5f06f-cab1-4546-a546-95fc619bce7b" containerName="payload-processing"
Nov 02 14:49:21.755186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.755147    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d5f06f-cab1-4546-a546-95fc619bce7b" containerName="payload-processing"
Nov 02 14:49:21.755284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.755274    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d157582e-3896-46f9-a053-010c45191ec3" containerName="istio-proxy"
Nov 02 14:49:21.755324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.755298    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d5f06f-cab1-4546-a546-95fc619bce7b" containerName="payload-processing"
Nov 02 14:49:21.759500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.759468    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:21.777966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.777938    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"]
Nov 02 14:49:21.786101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.786074    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:49:21.786231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.786161    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:49:21.827417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.827392    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:49:21.827574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.827471    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:49:21.850459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.850428    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0c8306-mxbpw\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:21.850601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.850470    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0c8306-mxbpw\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:21.850656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.850645    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qt65\" (UniqueName: \"kubernetes.io/projected/65d47e5f-b43d-43b9-bab0-4a71b7023f14-kube-api-access-4qt65\") pod \"maas-api-revoke-sub-cf0c8306-mxbpw\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:21.951877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.951840    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0c8306-mxbpw\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:21.952040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.951895    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0c8306-mxbpw\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:21.952097 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.952049    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qt65\" (UniqueName: \"kubernetes.io/projected/65d47e5f-b43d-43b9-bab0-4a71b7023f14-kube-api-access-4qt65\") pod \"maas-api-revoke-sub-cf0c8306-mxbpw\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:21.952661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.952637    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf0c8306-mxbpw\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:21.954379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.954360    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf0c8306-mxbpw\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:21.976729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:21.976706    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qt65\" (UniqueName: \"kubernetes.io/projected/65d47e5f-b43d-43b9-bab0-4a71b7023f14-kube-api-access-4qt65\") pod \"maas-api-revoke-sub-cf0c8306-mxbpw\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:22.042108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:22.042042    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"]
Nov 02 14:49:22.046244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:22.046217    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0610595-f9wrn"]
Nov 02 14:49:22.074614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:22.074589    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:22.386419 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:49:22.386389    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d47e5f_b43d_43b9_bab0_4a71b7023f14.slice/crio-aad6c7b66d3b6bf09171fdbb8d94a53fb316771f5f8c5a4fad34106094136c24 WatchSource:0}: Error finding container aad6c7b66d3b6bf09171fdbb8d94a53fb316771f5f8c5a4fad34106094136c24: Status 404 returned error can't find the container with id aad6c7b66d3b6bf09171fdbb8d94a53fb316771f5f8c5a4fad34106094136c24
Nov 02 14:49:22.387947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:22.387921    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"]
Nov 02 14:49:23.012500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:23.012458    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw" event={"ID":"65d47e5f-b43d-43b9-bab0-4a71b7023f14","Type":"ContainerStarted","Data":"808ba1e7ede9ff0a38e3d64dde3106a78d239dd88e3486c9c90b7b7b54b0bad9"}
Nov 02 14:49:23.013132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:23.012522    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw" event={"ID":"65d47e5f-b43d-43b9-bab0-4a71b7023f14","Type":"ContainerStarted","Data":"aad6c7b66d3b6bf09171fdbb8d94a53fb316771f5f8c5a4fad34106094136c24"}
Nov 02 14:49:23.060182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:23.060133    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw" podStartSLOduration=2.060120075 podStartE2EDuration="2.060120075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:49:23.056972077 +0000 UTC m=+3953.808276514" watchObservedRunningTime="2026-11-02 14:49:23.060120075 +0000 UTC m=+3953.811424513"
Nov 02 14:49:23.402319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:23.402227    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8f1d18-dcf9-42b1-9bd6-eec60d16da37" path="/var/lib/kubelet/pods/8f8f1d18-dcf9-42b1-9bd6-eec60d16da37/volumes"
Nov 02 14:49:24.022562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:24.022524    5034 generic.go:356] "Generic (PLEG): container finished" podID="65d47e5f-b43d-43b9-bab0-4a71b7023f14" containerID="808ba1e7ede9ff0a38e3d64dde3106a78d239dd88e3486c9c90b7b7b54b0bad9" exitCode=0
Nov 02 14:49:24.023011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:24.022606    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw" event={"ID":"65d47e5f-b43d-43b9-bab0-4a71b7023f14","Type":"ContainerDied","Data":"808ba1e7ede9ff0a38e3d64dde3106a78d239dd88e3486c9c90b7b7b54b0bad9"}
Nov 02 14:49:24.895611 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:24.895571    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"]
Nov 02 14:49:24.895902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:24.895863    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy" containerID="cri-o://f1c075b8c6db1b7d1f153eeca21d5ba435e0feda4992bfc0c38e0af1573cd05d" gracePeriod=30
Nov 02 14:49:25.406022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.405995    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:25.506189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.506107    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-openshift-ca\") pod \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") "
Nov 02 14:49:25.506364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.506253    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-xks-ca\") pod \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") "
Nov 02 14:49:25.506544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.506521    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-openshift-ca" pod "65d47e5f-b43d-43b9-bab0-4a71b7023f14" (UID: "65d47e5f-b43d-43b9-bab0-4a71b7023f14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:49:25.506820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.506796    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d47e5f-b43d-43b9-bab0-4a71b7023f14-kube-api-access-4qt65\" (UniqueName: \"kubernetes.io/projected/65d47e5f-b43d-43b9-bab0-4a71b7023f14-kube-api-access-4qt65\") pod \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\" (UID: \"65d47e5f-b43d-43b9-bab0-4a71b7023f14\") "
Nov 02 14:49:25.507477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.507429    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:25.508660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.508634    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-xks-ca" pod "65d47e5f-b43d-43b9-bab0-4a71b7023f14" (UID: "65d47e5f-b43d-43b9-bab0-4a71b7023f14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:49:25.509483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.509459    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d47e5f-b43d-43b9-bab0-4a71b7023f14-kube-api-access-4qt65" pod "65d47e5f-b43d-43b9-bab0-4a71b7023f14" (UID: "65d47e5f-b43d-43b9-bab0-4a71b7023f14"). InnerVolumeSpecName "kube-api-access-4qt65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:49:25.608928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.608900    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65d47e5f-b43d-43b9-bab0-4a71b7023f14-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:25.608928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:25.608924    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qt65\" (UniqueName: \"kubernetes.io/projected/65d47e5f-b43d-43b9-bab0-4a71b7023f14-kube-api-access-4qt65\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:26.123384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:26.123346    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw" event={"ID":"65d47e5f-b43d-43b9-bab0-4a71b7023f14","Type":"ContainerDied","Data":"aad6c7b66d3b6bf09171fdbb8d94a53fb316771f5f8c5a4fad34106094136c24"}
Nov 02 14:49:26.123384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:26.123375    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"
Nov 02 14:49:26.123384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:26.123392    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad6c7b66d3b6bf09171fdbb8d94a53fb316771f5f8c5a4fad34106094136c24"
Nov 02 14:49:30.202178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.202147    5034 generic.go:356] "Generic (PLEG): container finished" podID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerID="f1c075b8c6db1b7d1f153eeca21d5ba435e0feda4992bfc0c38e0af1573cd05d" exitCode=0
Nov 02 14:49:30.202591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.202225    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" event={"ID":"e4e4ace3-b064-47cc-a5f0-031dac8babf2","Type":"ContainerDied","Data":"f1c075b8c6db1b7d1f153eeca21d5ba435e0feda4992bfc0c38e0af1573cd05d"}
Nov 02 14:49:30.202591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.202298    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h" event={"ID":"e4e4ace3-b064-47cc-a5f0-031dac8babf2","Type":"ContainerDied","Data":"fa0f00f494d86b27ea1daba568d071c04cc2417e3280a69ad94b6ccbef255add"}
Nov 02 14:49:30.202591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.202321    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa0f00f494d86b27ea1daba568d071c04cc2417e3280a69ad94b6ccbef255add"
Nov 02 14:49:30.202591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.202340    5034 scope.go:122] "RemoveContainer" containerID="c69652c3bf2220dcb0d0e57aeac6982ddada3f044746d82c0efbb0dfc3eb4ad2"
Nov 02 14:49:30.219034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.219012    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:49:30.295980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.295952    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-kube-api-access-dgkjf\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-kube-api-access-dgkjf\") pod \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") "
Nov 02 14:49:30.296146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296005    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-token\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-token\") pod \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") "
Nov 02 14:49:30.296146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296088    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-socket\") pod \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") "
Nov 02 14:49:30.296285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296182    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-credential-socket\") pod \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") "
Nov 02 14:49:30.296355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296290    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-certs\") pod \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") "
Nov 02 14:49:30.296355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296347    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-data\") pod \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") "
Nov 02 14:49:30.296462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296398    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-podinfo\") pod \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") "
Nov 02 14:49:30.296526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296495    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-envoy\") pod \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") "
Nov 02 14:49:30.296714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296537    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istiod-ca-cert\") pod \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\" (UID: \"e4e4ace3-b064-47cc-a5f0-031dac8babf2\") "
Nov 02 14:49:30.296714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296569    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-credential-socket" pod "e4e4ace3-b064-47cc-a5f0-031dac8babf2" (UID: "e4e4ace3-b064-47cc-a5f0-031dac8babf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:49:30.296714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296694    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-certs" pod "e4e4ace3-b064-47cc-a5f0-031dac8babf2" (UID: "e4e4ace3-b064-47cc-a5f0-031dac8babf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:49:30.297019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296933    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-socket" pod "e4e4ace3-b064-47cc-a5f0-031dac8babf2" (UID: "e4e4ace3-b064-47cc-a5f0-031dac8babf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:49:30.297019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.296948    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-data" pod "e4e4ace3-b064-47cc-a5f0-031dac8babf2" (UID: "e4e4ace3-b064-47cc-a5f0-031dac8babf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:49:30.297266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.297241    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istiod-ca-cert" pod "e4e4ace3-b064-47cc-a5f0-031dac8babf2" (UID: "e4e4ace3-b064-47cc-a5f0-031dac8babf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:49:30.297404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.297328    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:30.297404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.297353    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:30.297404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.297368    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:30.297404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.297382    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:30.299335 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.299308    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-kube-api-access-dgkjf" pod "e4e4ace3-b064-47cc-a5f0-031dac8babf2" (UID: "e4e4ace3-b064-47cc-a5f0-031dac8babf2"). InnerVolumeSpecName "kube-api-access-dgkjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:49:30.299445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.299385    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-podinfo" pod "e4e4ace3-b064-47cc-a5f0-031dac8babf2" (UID: "e4e4ace3-b064-47cc-a5f0-031dac8babf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:49:30.300009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.299985    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-token" pod "e4e4ace3-b064-47cc-a5f0-031dac8babf2" (UID: "e4e4ace3-b064-47cc-a5f0-031dac8babf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:49:30.300409 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.300385    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-envoy" pod "e4e4ace3-b064-47cc-a5f0-031dac8babf2" (UID: "e4e4ace3-b064-47cc-a5f0-031dac8babf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:49:30.398656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.398626    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:30.398656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.398653    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:30.398656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.398661    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:30.398921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.398669    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgkjf\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-kube-api-access-dgkjf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:30.398921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:30.398679    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4e4ace3-b064-47cc-a5f0-031dac8babf2-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:49:31.219820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:31.219587    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"
Nov 02 14:49:31.312952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:31.312884    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"]
Nov 02 14:49:31.321824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:31.321792    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c8f647-openshift-default-5b8c59fc75-n5s7h"]
Nov 02 14:49:31.397504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:31.397469    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" path="/var/lib/kubelet/pods/e4e4ace3-b064-47cc-a5f0-031dac8babf2/volumes"
Nov 02 14:49:38.880803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:38.879351    5034 scope.go:122] "RemoveContainer" containerID="0f5b02fc3d4b8a0fd561addf7c17321528ad0349a81b74ec4e266cef703388bc"
Nov 02 14:49:38.943760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:38.943736    5034 scope.go:122] "RemoveContainer" containerID="f1c075b8c6db1b7d1f153eeca21d5ba435e0feda4992bfc0c38e0af1573cd05d"
Nov 02 14:49:48.416674 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:49:48.416635    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:49:55.758221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.758187    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"]
Nov 02 14:49:55.759242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.759216    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy"
Nov 02 14:49:55.759242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.759241    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy"
Nov 02 14:49:55.759450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.759261    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy"
Nov 02 14:49:55.759450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.759269    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy"
Nov 02 14:49:55.759450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.759289    5034 cpu_manager.go:414] "containerMap: removing container" podUID="65d47e5f-b43d-43b9-bab0-4a71b7023f14" containerName="revoke-keys"
Nov 02 14:49:55.759450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.759295    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d47e5f-b43d-43b9-bab0-4a71b7023f14" containerName="revoke-keys"
Nov 02 14:49:55.759664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.759527    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy"
Nov 02 14:49:55.759664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.759547    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e4ace3-b064-47cc-a5f0-031dac8babf2" containerName="istio-proxy"
Nov 02 14:49:55.759664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.759559    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d47e5f-b43d-43b9-bab0-4a71b7023f14" containerName="revoke-keys"
Nov 02 14:49:55.765271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.765248    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.792303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.792274    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"]
Nov 02 14:49:55.853555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.853518    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldcg6\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-kube-api-access-ldcg6\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.853752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.853642    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.853752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.853729    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.853991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.853834    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.853991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.853899    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.853991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.853953    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.854140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.854118    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.854197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.854169    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.854310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.854294    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955155    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955241    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955360    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldcg6\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-kube-api-access-ldcg6\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955418    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955475    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955511    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955569    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955609    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955708    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.955866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.955846    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.956032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.956000    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.956112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.956033    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.956265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.956241    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.956503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.956483    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.957946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.957914    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.957946 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.957932    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.980883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.980856    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:55.981607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:55.981270    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldcg6\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-kube-api-access-ldcg6\") pod \"maas-default-gateway-openshift-default-6886b96f9-vgfz8\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:56.080287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:56.080206    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:56.439275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:56.439241    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:49:56.439401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:56.439312    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:49:56.439401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:56.439341    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:49:56.440914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:56.440889    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"]
Nov 02 14:49:56.564821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:56.557033    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" event={"ID":"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49","Type":"ContainerStarted","Data":"60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57"}
Nov 02 14:49:56.564821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:56.557096    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" event={"ID":"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49","Type":"ContainerStarted","Data":"26a0b4c5226a74056b5dfb989b8be738222c520b43d97aa9dab7bce6fd154d14"}
Nov 02 14:49:56.628235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:56.628184    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podStartSLOduration=1.628168321 podStartE2EDuration="1.628168321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:49:56.626773589 +0000 UTC m=+3987.378078026" watchObservedRunningTime="2026-11-02 14:49:56.628168321 +0000 UTC m=+3987.379472759"
Nov 02 14:49:57.080503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:57.080470    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:49:57.081792 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:57.081749    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vgfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body=
Nov 02 14:49:57.081934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:57.081833    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused"
Nov 02 14:49:58.087793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:58.083893    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vgfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body=
Nov 02 14:49:58.087793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:58.083963    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused"
Nov 02 14:49:59.088802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:59.087096    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vgfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body=
Nov 02 14:49:59.088802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:49:59.087159    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused"
Nov 02 14:50:00.137052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:00.136354    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vgfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body=
Nov 02 14:50:00.137052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:00.136410    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused"
Nov 02 14:50:01.085800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:01.081799    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vgfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body=
Nov 02 14:50:01.085800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:01.081867    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused"
Nov 02 14:50:02.082894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:02.082858    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vgfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body=
Nov 02 14:50:02.083522 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:02.083478    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused"
Nov 02 14:50:03.080834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:03.080794    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vgfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body=
Nov 02 14:50:03.081051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:03.080865    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused"
Nov 02 14:50:04.081257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:04.081220    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-vgfz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused" start-of-body=
Nov 02 14:50:04.081697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:04.081296    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.151:15021/healthz/ready\": dial tcp 10.132.1.151:15021: connect: connection refused"
Nov 02 14:50:05.140002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:05.139963    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:50:05.140457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:05.140435    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:50:05.141215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:05.141193    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:50:12.339857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:12.334865    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:50:12.339857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:12.335325    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded"
Nov 02 14:50:12.438801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:12.437850    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:50:17.149430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:17.149392    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"]
Nov 02 14:50:17.150200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:17.150148    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerName="istio-proxy" containerID="cri-o://83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a" gracePeriod=30
Nov 02 14:50:18.415828 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:50:18.415772    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:50:22.552885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.547013    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:50:22.646820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.646765    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-kube-api-access-9sqjc\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-kube-api-access-9sqjc\") pod \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") "
Nov 02 14:50:22.647015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.646917    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-podinfo\") pod \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") "
Nov 02 14:50:22.647015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.646962    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-socket\") pod \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") "
Nov 02 14:50:22.647176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.647076    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-envoy\") pod \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") "
Nov 02 14:50:22.647176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.647109    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istiod-ca-cert\") pod \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") "
Nov 02 14:50:22.647176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.647161    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-certs\") pod \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") "
Nov 02 14:50:22.647350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.647204    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-credential-socket\") pod \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") "
Nov 02 14:50:22.647350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.647275    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-socket" pod "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" (UID: "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:22.647350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.647339    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-token\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-token\") pod \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") "
Nov 02 14:50:22.647523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.647373    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-data\") pod \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\" (UID: \"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3\") "
Nov 02 14:50:22.648148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.647576    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istiod-ca-cert" pod "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" (UID: "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:50:22.648148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.648078    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:22.648148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.648100    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:22.649038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.648742    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-data" pod "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" (UID: "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:22.649038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.648823    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-credential-socket" pod "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" (UID: "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:22.649385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.649364    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-certs" pod "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" (UID: "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:22.649905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.649878    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-podinfo" pod "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" (UID: "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:50:22.650032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.650007    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-envoy" pod "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" (UID: "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:50:22.650978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.650953    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-kube-api-access-9sqjc" pod "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" (UID: "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3"). InnerVolumeSpecName "kube-api-access-9sqjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:22.651296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.651270    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-token" pod "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" (UID: "70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:22.749402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.749372    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sqjc\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-kube-api-access-9sqjc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:22.749402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.749400    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:22.749601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.749410    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:22.749601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.749418    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:22.749601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.749426    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:22.749601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.749435    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:22.749601 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.749443    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:22.966457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.966420    5034 generic.go:356] "Generic (PLEG): container finished" podID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerID="83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a" exitCode=0
Nov 02 14:50:22.966673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.966495    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"
Nov 02 14:50:22.966673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.966503    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" event={"ID":"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3","Type":"ContainerDied","Data":"83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a"}
Nov 02 14:50:22.966673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.966638    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6" event={"ID":"70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3","Type":"ContainerDied","Data":"2bfbde355284e0690b4dcadb1c7c7968965324ea6c5a05eac2c48bdfb12c0f95"}
Nov 02 14:50:22.966993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.966691    5034 scope.go:122] "RemoveContainer" containerID="83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a"
Nov 02 14:50:22.985274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.985249    5034 scope.go:122] "RemoveContainer" containerID="83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a"
Nov 02 14:50:22.985590 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:50:22.985563    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a\": container with ID starting with 83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a not found: ID does not exist" containerID="83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a"
Nov 02 14:50:22.985687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:22.985595    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a"} err="failed to get container status \"83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a\": rpc error: code = NotFound desc = could not find container \"83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a\": container with ID starting with 83a25e64533362bd1a03d43489e36d15f78142ee844b3b953d134b349494ca7a not found: ID does not exist"
Nov 02 14:50:23.028733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:23.028666    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"]
Nov 02 14:50:23.033173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:23.033150    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-80133f-openshift-default-69f568b6c9-b9pb6"]
Nov 02 14:50:23.397714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:23.397626    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" path="/var/lib/kubelet/pods/70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3/volumes"
Nov 02 14:50:34.315305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:34.315266    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"]
Nov 02 14:50:34.354899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:34.354868    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"]
Nov 02 14:50:39.034034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:39.033997    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"]
Nov 02 14:50:39.044572 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:39.044540    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54fe25ac-rdgpq"]
Nov 02 14:50:39.166415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:39.166385    5034 scope.go:122] "RemoveContainer" containerID="903c3abce5f4413f9244fac825fb2471b1a8487c69dc54b277ea7ffb90b24ab5"
Nov 02 14:50:39.180277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:39.180249    5034 scope.go:122] "RemoveContainer" containerID="ab0ec1b1dada09de38895c8c7ae77838ed464180cc3967dd3cec057f6d525f86"
Nov 02 14:50:39.193565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:39.193540    5034 scope.go:122] "RemoveContainer" containerID="110bdc90c3ca03fae53c30c36e91c41a2f671655356ea0a4bd7b5f1600493d36"
Nov 02 14:50:39.205355 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:39.205327    5034 scope.go:122] "RemoveContainer" containerID="5e9cc74d55c57842a626092192d85e79dfa711db99e81d8bfc817ae4a83e49f9"
Nov 02 14:50:39.315873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:39.315763    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb" podUID="cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" containerName="payload-processing" containerID="cri-o://8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd" gracePeriod=25
Nov 02 14:50:39.355840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:39.355771    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql" podUID="ccee94a9-7644-45ab-968a-c08d8793e67e" containerName="payload-pre-processing" containerID="cri-o://3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650" gracePeriod=25
Nov 02 14:50:39.401474 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:39.401442    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9616290c-1ed2-450d-8532-525e9d9eb7fd" path="/var/lib/kubelet/pods/9616290c-1ed2-450d-8532-525e9d9eb7fd/volumes"
Nov 02 14:50:40.696555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:40.696507    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql" podUID="ccee94a9-7644-45ab-968a-c08d8793e67e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:50:41.704100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:41.704052    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb" podUID="cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:50:42.576965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:42.576020    5034 patch_prober.go:29] interesting pod/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": read tcp 10.132.0.2:52356->10.132.1.149:15021: read: connection reset by peer" start-of-body=
Nov 02 14:50:42.576965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:42.576078    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": read tcp 10.132.0.2:52356->10.132.1.149:15021: read: connection reset by peer"
Nov 02 14:50:43.219476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:43.219442    5034 generic.go:356] "Generic (PLEG): container finished" podID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerID="6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c" exitCode=0
Nov 02 14:50:43.220002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:43.219511    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" event={"ID":"be2c7637-7247-4cf6-a551-02f2fc51c21c","Type":"ContainerDied","Data":"6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c"}
Nov 02 14:50:43.220077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:43.220068    5034 scope.go:122] "RemoveContainer" containerID="6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c"
Nov 02 14:50:43.222508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:43.222475    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:50:43.222625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:43.222545    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:50:43.222625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:43.222577    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:50:44.230225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:44.230194    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" event={"ID":"be2c7637-7247-4cf6-a551-02f2fc51c21c","Type":"ContainerStarted","Data":"88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d"}
Nov 02 14:50:45.230739 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:45.230698    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:50:45.231197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:45.231026    5034 patch_prober.go:29] interesting pod/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Nov 02 14:50:45.231197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:45.231094    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Nov 02 14:50:46.231532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:46.231497    5034 patch_prober.go:29] interesting pod/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Nov 02 14:50:46.231968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:46.231550    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Nov 02 14:50:47.231431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:47.231388    5034 patch_prober.go:29] interesting pod/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Nov 02 14:50:47.231614 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:47.231451    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Nov 02 14:50:48.231545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:48.231505    5034 patch_prober.go:29] interesting pod/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Nov 02 14:50:48.231759 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:48.231563    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Nov 02 14:50:48.428493 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:50:48.428423    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:50:49.231294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:49.231258    5034 patch_prober.go:29] interesting pod/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused" start-of-body=
Nov 02 14:50:49.231492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:49.231315    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.149:15021/healthz/ready\": dial tcp 10.132.1.149:15021: connect: connection refused"
Nov 02 14:50:50.040975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:50.040937    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"]
Nov 02 14:50:50.053588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:50.053557    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1045bfbd-kx9pj"]
Nov 02 14:50:50.235107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:50.235077    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:50:50.235396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:50.235369    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:50:50.236052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:50.236029    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:50:50.696001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:50.695960    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql" podUID="ccee94a9-7644-45ab-968a-c08d8793e67e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:50:51.396492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:51.396458    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc418012-6613-46f8-b646-afb5cd9a6bca" path="/var/lib/kubelet/pods/bc418012-6613-46f8-b646-afb5cd9a6bca/volumes"
Nov 02 14:50:51.703978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:51.703892    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb" podUID="cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:50:59.688180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.688153    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:50:59.691704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.691685    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:50:59.781609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.781573    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccee94a9-7644-45ab-968a-c08d8793e67e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee94a9-7644-45ab-968a-c08d8793e67e-plugins-config-volume\") pod \"ccee94a9-7644-45ab-968a-c08d8793e67e\" (UID: \"ccee94a9-7644-45ab-968a-c08d8793e67e\") "
Nov 02 14:50:59.781821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.781691    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccee94a9-7644-45ab-968a-c08d8793e67e-kube-api-access-c4hdv\" (UniqueName: \"kubernetes.io/projected/ccee94a9-7644-45ab-968a-c08d8793e67e-kube-api-access-c4hdv\") pod \"ccee94a9-7644-45ab-968a-c08d8793e67e\" (UID: \"ccee94a9-7644-45ab-968a-c08d8793e67e\") "
Nov 02 14:50:59.781953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.781925    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-kube-api-access-t96sd\" (UniqueName: \"kubernetes.io/projected/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-kube-api-access-t96sd\") pod \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\" (UID: \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\") "
Nov 02 14:50:59.782024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.781968    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-plugins-config-volume\") pod \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\" (UID: \"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9\") "
Nov 02 14:50:59.784156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.782983    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-plugins-config-volume" pod "cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" (UID: "cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:50:59.784827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.784801    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccee94a9-7644-45ab-968a-c08d8793e67e-plugins-config-volume" pod "ccee94a9-7644-45ab-968a-c08d8793e67e" (UID: "ccee94a9-7644-45ab-968a-c08d8793e67e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:50:59.789035 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.788995    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-kube-api-access-t96sd" pod "cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" (UID: "cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9"). InnerVolumeSpecName "kube-api-access-t96sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:59.791195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.791168    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccee94a9-7644-45ab-968a-c08d8793e67e-kube-api-access-c4hdv" pod "ccee94a9-7644-45ab-968a-c08d8793e67e" (UID: "ccee94a9-7644-45ab-968a-c08d8793e67e"). InnerVolumeSpecName "kube-api-access-c4hdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:50:59.884447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.884413    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t96sd\" (UniqueName: \"kubernetes.io/projected/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-kube-api-access-t96sd\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:59.884447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.884449    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:59.884643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.884463    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccee94a9-7644-45ab-968a-c08d8793e67e-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:50:59.884643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:50:59.884476    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4hdv\" (UniqueName: \"kubernetes.io/projected/ccee94a9-7644-45ab-968a-c08d8793e67e-kube-api-access-c4hdv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:51:00.375284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.375251    5034 generic.go:356] "Generic (PLEG): container finished" podID="cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" containerID="8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd" exitCode=0
Nov 02 14:51:00.375476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.375302    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"
Nov 02 14:51:00.375476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.375327    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb" event={"ID":"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9","Type":"ContainerDied","Data":"8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd"}
Nov 02 14:51:00.375476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.375380    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb" event={"ID":"cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9","Type":"ContainerDied","Data":"ac9c385b37693c0483886e23630bbde52ee4758deeb855d2e8f8df7127bd59d4"}
Nov 02 14:51:00.375476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.375406    5034 scope.go:122] "RemoveContainer" containerID="8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd"
Nov 02 14:51:00.377039 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.377016    5034 generic.go:356] "Generic (PLEG): container finished" podID="ccee94a9-7644-45ab-968a-c08d8793e67e" containerID="3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650" exitCode=0
Nov 02 14:51:00.377174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.377054    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"
Nov 02 14:51:00.377174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.377061    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql" event={"ID":"ccee94a9-7644-45ab-968a-c08d8793e67e","Type":"ContainerDied","Data":"3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650"}
Nov 02 14:51:00.377174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.377100    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql" event={"ID":"ccee94a9-7644-45ab-968a-c08d8793e67e","Type":"ContainerDied","Data":"06b996497dac25157aa60bec23768ddc0f1696063c35d99e1bb82de5cbfadfe5"}
Nov 02 14:51:00.389849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.389830    5034 scope.go:122] "RemoveContainer" containerID="8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd"
Nov 02 14:51:00.390104 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:51:00.390081    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd\": container with ID starting with 8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd not found: ID does not exist" containerID="8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd"
Nov 02 14:51:00.390164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.390110    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd"} err="failed to get container status \"8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd\": rpc error: code = NotFound desc = could not find container \"8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd\": container with ID starting with 8a247c98bab7ad463c799e6455d75a04b9fd3a2448298ba9e14f5e4af59b20fd not found: ID does not exist"
Nov 02 14:51:00.390164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.390125    5034 scope.go:122] "RemoveContainer" containerID="3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650"
Nov 02 14:51:00.400846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.400829    5034 scope.go:122] "RemoveContainer" containerID="3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650"
Nov 02 14:51:00.401082 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:51:00.401065    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650\": container with ID starting with 3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650 not found: ID does not exist" containerID="3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650"
Nov 02 14:51:00.401131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.401089    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650"} err="failed to get container status \"3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650\": rpc error: code = NotFound desc = could not find container \"3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650\": container with ID starting with 3ac3fb51be67c848203a8815e8a11ce2e39cc4e78725ee0508258f8eb5268650 not found: ID does not exist"
Nov 02 14:51:00.441988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.441966    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"]
Nov 02 14:51:00.461382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.461357    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-87efd3-5c88f57567-dlpql"]
Nov 02 14:51:00.499142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.499114    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"]
Nov 02 14:51:00.503866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:00.503846    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-87efd3-5cd98f5c69-lj6pb"]
Nov 02 14:51:01.396962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:01.396931    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" path="/var/lib/kubelet/pods/cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9/volumes"
Nov 02 14:51:01.397415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:01.397401    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccee94a9-7644-45ab-968a-c08d8793e67e" path="/var/lib/kubelet/pods/ccee94a9-7644-45ab-968a-c08d8793e67e/volumes"
Nov 02 14:51:02.863446 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.863417    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"]
Nov 02 14:51:02.864256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.864239    5034 cpu_manager.go:414] "containerMap: removing container" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerName="istio-proxy"
Nov 02 14:51:02.864256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.864257    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerName="istio-proxy"
Nov 02 14:51:02.864348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.864271    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" containerName="payload-processing"
Nov 02 14:51:02.864348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.864279    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" containerName="payload-processing"
Nov 02 14:51:02.864348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.864295    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ccee94a9-7644-45ab-968a-c08d8793e67e" containerName="payload-pre-processing"
Nov 02 14:51:02.864348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.864300    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccee94a9-7644-45ab-968a-c08d8793e67e" containerName="payload-pre-processing"
Nov 02 14:51:02.864480 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.864474    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="70178c0a-4f8a-49ab-8cb5-d565c3d3b9d3" containerName="istio-proxy"
Nov 02 14:51:02.864514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.864484    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc650161-5d67-41d9-b9b4-2e7c1bfaaeb9" containerName="payload-processing"
Nov 02 14:51:02.864514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.864491    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccee94a9-7644-45ab-968a-c08d8793e67e" containerName="payload-pre-processing"
Nov 02 14:51:02.870194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.870174    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:02.888554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.888525    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"]
Nov 02 14:51:02.895661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.895635    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:51:02.895850 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.895673    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:51:02.895969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.895769    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:51:02.896072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:02.896045    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:51:03.031531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.031497    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ltx\" (UniqueName: \"kubernetes.io/projected/3bf2886c-a17e-4255-a8be-5790ddacd3fd-kube-api-access-v8ltx\") pod \"maas-api-revoke-sub-2d608aec-h62gs\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.031731 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.031709    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d608aec-h62gs\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.031800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.031743    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d608aec-h62gs\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.133278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.133205    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ltx\" (UniqueName: \"kubernetes.io/projected/3bf2886c-a17e-4255-a8be-5790ddacd3fd-kube-api-access-v8ltx\") pod \"maas-api-revoke-sub-2d608aec-h62gs\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.133424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.133396    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d608aec-h62gs\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.133473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.133427    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d608aec-h62gs\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.134095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.134061    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d608aec-h62gs\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.135923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.135904    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d608aec-h62gs\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.158314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.158283    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ltx\" (UniqueName: \"kubernetes.io/projected/3bf2886c-a17e-4255-a8be-5790ddacd3fd-kube-api-access-v8ltx\") pod \"maas-api-revoke-sub-2d608aec-h62gs\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.185807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.185766    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:03.495400 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:51:03.495357    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf2886c_a17e_4255_a8be_5790ddacd3fd.slice/crio-d452942c16debd8865670af5a6dc2e62459e73caaf39c532d2b689bcff7c6676 WatchSource:0}: Error finding container d452942c16debd8865670af5a6dc2e62459e73caaf39c532d2b689bcff7c6676: Status 404 returned error can't find the container with id d452942c16debd8865670af5a6dc2e62459e73caaf39c532d2b689bcff7c6676
Nov 02 14:51:03.499297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:03.499266    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"]
Nov 02 14:51:04.419101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:04.419072    5034 generic.go:356] "Generic (PLEG): container finished" podID="3bf2886c-a17e-4255-a8be-5790ddacd3fd" containerID="587c480e73d1a59cb0fe36cefb2d304fadc9a7335733563fc03bea80ff23d950" exitCode=0
Nov 02 14:51:04.419410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:04.419158    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs" event={"ID":"3bf2886c-a17e-4255-a8be-5790ddacd3fd","Type":"ContainerDied","Data":"587c480e73d1a59cb0fe36cefb2d304fadc9a7335733563fc03bea80ff23d950"}
Nov 02 14:51:04.419410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:04.419209    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs" event={"ID":"3bf2886c-a17e-4255-a8be-5790ddacd3fd","Type":"ContainerStarted","Data":"d452942c16debd8865670af5a6dc2e62459e73caaf39c532d2b689bcff7c6676"}
Nov 02 14:51:05.665302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.665273    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:05.772906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.772876    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf2886c-a17e-4255-a8be-5790ddacd3fd-kube-api-access-v8ltx\" (UniqueName: \"kubernetes.io/projected/3bf2886c-a17e-4255-a8be-5790ddacd3fd-kube-api-access-v8ltx\") pod \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") "
Nov 02 14:51:05.773089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.772924    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-xks-ca\") pod \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") "
Nov 02 14:51:05.773285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.773263    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-openshift-ca\") pod \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\" (UID: \"3bf2886c-a17e-4255-a8be-5790ddacd3fd\") "
Nov 02 14:51:05.773680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.773652    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-openshift-ca" pod "3bf2886c-a17e-4255-a8be-5790ddacd3fd" (UID: "3bf2886c-a17e-4255-a8be-5790ddacd3fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:51:05.775211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.775185    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf2886c-a17e-4255-a8be-5790ddacd3fd-kube-api-access-v8ltx" pod "3bf2886c-a17e-4255-a8be-5790ddacd3fd" (UID: "3bf2886c-a17e-4255-a8be-5790ddacd3fd"). InnerVolumeSpecName "kube-api-access-v8ltx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:51:05.775521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.775497    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-xks-ca" pod "3bf2886c-a17e-4255-a8be-5790ddacd3fd" (UID: "3bf2886c-a17e-4255-a8be-5790ddacd3fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:51:05.874982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.874951    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:51:05.874982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.874980    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ltx\" (UniqueName: \"kubernetes.io/projected/3bf2886c-a17e-4255-a8be-5790ddacd3fd-kube-api-access-v8ltx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:51:05.875138 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:05.874995    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf2886c-a17e-4255-a8be-5790ddacd3fd-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:51:06.436503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:06.436478    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"
Nov 02 14:51:06.436693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:06.436477    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs" event={"ID":"3bf2886c-a17e-4255-a8be-5790ddacd3fd","Type":"ContainerDied","Data":"d452942c16debd8865670af5a6dc2e62459e73caaf39c532d2b689bcff7c6676"}
Nov 02 14:51:06.436693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:06.436579    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d452942c16debd8865670af5a6dc2e62459e73caaf39c532d2b689bcff7c6676"
Nov 02 14:51:18.419992 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:51:18.419952    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:51:24.817665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:24.817630    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"]
Nov 02 14:51:29.818785 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:29.818730    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h" podUID="f28a0afb-2741-4a34-ab83-ff5af083c1a8" containerName="payload-pre-processing" containerID="cri-o://84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2" gracePeriod=25
Nov 02 14:51:33.050396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:33.050355    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h" podUID="f28a0afb-2741-4a34-ab83-ff5af083c1a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:51:39.266383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:39.266348    5034 scope.go:122] "RemoveContainer" containerID="3b49b59fde8558b0faf734b520789db6260e0ab08e7ad65c4264ea29e3b562b0"
Nov 02 14:51:43.051770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:43.051667    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h" podUID="f28a0afb-2741-4a34-ab83-ff5af083c1a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:51:48.416250 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:51:48.416211    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:51:50.142585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.142555    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:51:50.145615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.145584    5034 generic.go:356] "Generic (PLEG): container finished" podID="f28a0afb-2741-4a34-ab83-ff5af083c1a8" containerID="84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2" exitCode=0
Nov 02 14:51:50.145877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.145854    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h" event={"ID":"f28a0afb-2741-4a34-ab83-ff5af083c1a8","Type":"ContainerDied","Data":"84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2"}
Nov 02 14:51:50.145961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.145915    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h" event={"ID":"f28a0afb-2741-4a34-ab83-ff5af083c1a8","Type":"ContainerDied","Data":"270513a627d07e9d8e1ddd0d58eefabfdf937c61a31edf58011e1e2a00899450"}
Nov 02 14:51:50.145961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.145948    5034 scope.go:122] "RemoveContainer" containerID="84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2"
Nov 02 14:51:50.146120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.146104    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"
Nov 02 14:51:50.163152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.163070    5034 scope.go:122] "RemoveContainer" containerID="84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2"
Nov 02 14:51:50.163600 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:51:50.163568    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2\": container with ID starting with 84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2 not found: ID does not exist" containerID="84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2"
Nov 02 14:51:50.163688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.163611    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2"} err="failed to get container status \"84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2\": rpc error: code = NotFound desc = could not find container \"84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2\": container with ID starting with 84d96bf4b521b304cefd99d43f3f97abe433e125fd59fd64e69a5f4bdca460c2 not found: ID does not exist"
Nov 02 14:51:50.239660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.239623    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28a0afb-2741-4a34-ab83-ff5af083c1a8-kube-api-access-6grp8\" (UniqueName: \"kubernetes.io/projected/f28a0afb-2741-4a34-ab83-ff5af083c1a8-kube-api-access-6grp8\") pod \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\" (UID: \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\") "
Nov 02 14:51:50.240165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.240140    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28a0afb-2741-4a34-ab83-ff5af083c1a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28a0afb-2741-4a34-ab83-ff5af083c1a8-plugins-config-volume\") pod \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\" (UID: \"f28a0afb-2741-4a34-ab83-ff5af083c1a8\") "
Nov 02 14:51:50.241689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.241664    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28a0afb-2741-4a34-ab83-ff5af083c1a8-plugins-config-volume" pod "f28a0afb-2741-4a34-ab83-ff5af083c1a8" (UID: "f28a0afb-2741-4a34-ab83-ff5af083c1a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:51:50.249685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.249654    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28a0afb-2741-4a34-ab83-ff5af083c1a8-kube-api-access-6grp8" pod "f28a0afb-2741-4a34-ab83-ff5af083c1a8" (UID: "f28a0afb-2741-4a34-ab83-ff5af083c1a8"). InnerVolumeSpecName "kube-api-access-6grp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:51:50.342491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.342456    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6grp8\" (UniqueName: \"kubernetes.io/projected/f28a0afb-2741-4a34-ab83-ff5af083c1a8-kube-api-access-6grp8\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:51:50.342491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.342487    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28a0afb-2741-4a34-ab83-ff5af083c1a8-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:51:50.498141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.498110    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"]
Nov 02 14:51:50.502650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:50.502619    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bc34db-68c766bcbc-9rh9h"]
Nov 02 14:51:51.397407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:51.397375    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28a0afb-2741-4a34-ab83-ff5af083c1a8" path="/var/lib/kubelet/pods/f28a0afb-2741-4a34-ab83-ff5af083c1a8/volumes"
Nov 02 14:51:53.031036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:53.031002    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"]
Nov 02 14:51:53.044623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:53.044591    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1ac6bc3-dj45n"]
Nov 02 14:51:53.396631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:51:53.396549    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7ed36b-7186-452d-a777-a449b205ab66" path="/var/lib/kubelet/pods/0a7ed36b-7186-452d-a777-a449b205ab66/volumes"
Nov 02 14:52:04.080190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:04.080151    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"]
Nov 02 14:52:04.092727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:04.092695    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a74376b-tprx4"]
Nov 02 14:52:05.396941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:05.396909    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86406096-39d6-494a-87ec-c720e81f9c8b" path="/var/lib/kubelet/pods/86406096-39d6-494a-87ec-c720e81f9c8b/volumes"
Nov 02 14:52:10.087888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:10.087846    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"]
Nov 02 14:52:10.088433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:10.088112    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy" containerID="cri-o://945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09" gracePeriod=30
Nov 02 14:52:15.382023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.381963    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:52:15.436676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.436649    5034 generic.go:356] "Generic (PLEG): container finished" podID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerID="945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09" exitCode=0
Nov 02 14:52:15.436860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.436697    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"
Nov 02 14:52:15.436860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.436734    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" event={"ID":"3f7c5551-d472-445c-b35f-b938c5dab57c","Type":"ContainerDied","Data":"945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09"}
Nov 02 14:52:15.436860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.436820    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5" event={"ID":"3f7c5551-d472-445c-b35f-b938c5dab57c","Type":"ContainerDied","Data":"5ee697b79abf65571a8a8336fa21e383da139fec9ccb2bcb9630f6a4305acd41"}
Nov 02 14:52:15.436860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.436853    5034 scope.go:122] "RemoveContainer" containerID="945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09"
Nov 02 14:52:15.448526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.448508    5034 scope.go:122] "RemoveContainer" containerID="e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755"
Nov 02 14:52:15.459512 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.459488    5034 scope.go:122] "RemoveContainer" containerID="945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09"
Nov 02 14:52:15.459845 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:15.459817    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09\": container with ID starting with 945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09 not found: ID does not exist" containerID="945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09"
Nov 02 14:52:15.459929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.459855    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09"} err="failed to get container status \"945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09\": rpc error: code = NotFound desc = could not find container \"945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09\": container with ID starting with 945b7dfe9ee002fe441fdd23ce6720458d921b10388d3cbfb9a209210adeff09 not found: ID does not exist"
Nov 02 14:52:15.459929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.459879    5034 scope.go:122] "RemoveContainer" containerID="e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755"
Nov 02 14:52:15.460174 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:15.460155    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755\": container with ID starting with e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755 not found: ID does not exist" containerID="e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755"
Nov 02 14:52:15.460225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.460181    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755"} err="failed to get container status \"e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755\": rpc error: code = NotFound desc = could not find container \"e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755\": container with ID starting with e8c1be28ef49998178ec83320183aa67f5823491c6653baee17c8e4dbd49a755 not found: ID does not exist"
Nov 02 14:52:15.497160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.497124    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-token\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-token\") pod \"3f7c5551-d472-445c-b35f-b938c5dab57c\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") "
Nov 02 14:52:15.497308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.497202    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-socket\") pod \"3f7c5551-d472-445c-b35f-b938c5dab57c\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") "
Nov 02 14:52:15.497308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.497261    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-credential-socket\") pod \"3f7c5551-d472-445c-b35f-b938c5dab57c\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") "
Nov 02 14:52:15.497308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.497301    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7c5551-d472-445c-b35f-b938c5dab57c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7c5551-d472-445c-b35f-b938c5dab57c-istiod-ca-cert\") pod \"3f7c5551-d472-445c-b35f-b938c5dab57c\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") "
Nov 02 14:52:15.497490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.497369    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-data\") pod \"3f7c5551-d472-445c-b35f-b938c5dab57c\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") "
Nov 02 14:52:15.497490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.497421    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-certs\") pod \"3f7c5551-d472-445c-b35f-b938c5dab57c\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") "
Nov 02 14:52:15.497490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.497449    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-kube-api-access-kwvrx\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-kube-api-access-kwvrx\") pod \"3f7c5551-d472-445c-b35f-b938c5dab57c\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") "
Nov 02 14:52:15.497657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.497579    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-envoy\") pod \"3f7c5551-d472-445c-b35f-b938c5dab57c\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") "
Nov 02 14:52:15.497657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.497639    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-podinfo\") pod \"3f7c5551-d472-445c-b35f-b938c5dab57c\" (UID: \"3f7c5551-d472-445c-b35f-b938c5dab57c\") "
Nov 02 14:52:15.503743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.499766    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7c5551-d472-445c-b35f-b938c5dab57c-istiod-ca-cert" pod "3f7c5551-d472-445c-b35f-b938c5dab57c" (UID: "3f7c5551-d472-445c-b35f-b938c5dab57c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:15.503743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.500048    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-socket" pod "3f7c5551-d472-445c-b35f-b938c5dab57c" (UID: "3f7c5551-d472-445c-b35f-b938c5dab57c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:15.503743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.500251    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-credential-socket" pod "3f7c5551-d472-445c-b35f-b938c5dab57c" (UID: "3f7c5551-d472-445c-b35f-b938c5dab57c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:15.503743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.501632    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-token" pod "3f7c5551-d472-445c-b35f-b938c5dab57c" (UID: "3f7c5551-d472-445c-b35f-b938c5dab57c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:15.503743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.501886    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-data" pod "3f7c5551-d472-445c-b35f-b938c5dab57c" (UID: "3f7c5551-d472-445c-b35f-b938c5dab57c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:15.503743 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.502080    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-certs" pod "3f7c5551-d472-445c-b35f-b938c5dab57c" (UID: "3f7c5551-d472-445c-b35f-b938c5dab57c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:15.506878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.506849    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-kube-api-access-kwvrx" pod "3f7c5551-d472-445c-b35f-b938c5dab57c" (UID: "3f7c5551-d472-445c-b35f-b938c5dab57c"). InnerVolumeSpecName "kube-api-access-kwvrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:15.506878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.506862    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-podinfo" pod "3f7c5551-d472-445c-b35f-b938c5dab57c" (UID: "3f7c5551-d472-445c-b35f-b938c5dab57c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:52:15.507078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.507054    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-envoy" pod "3f7c5551-d472-445c-b35f-b938c5dab57c" (UID: "3f7c5551-d472-445c-b35f-b938c5dab57c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:15.601839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.601801    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:15.602097 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.602080    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwvrx\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-kube-api-access-kwvrx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:15.602228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.602211    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:15.602360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.602342    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:15.602462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.602453    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:15.602549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.602540    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:15.602665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.602645    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:15.602793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.602764    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7c5551-d472-445c-b35f-b938c5dab57c-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:15.602917 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.602903    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7c5551-d472-445c-b35f-b938c5dab57c-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:15.791617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.791584    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"]
Nov 02 14:52:15.802243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:15.802213    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2d13f4-openshift-default-795f5974f7-tdwj5"]
Nov 02 14:52:16.661876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:16.661833    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"]
Nov 02 14:52:16.662378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:16.662064    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy" containerID="cri-o://319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf" gracePeriod=30
Nov 02 14:52:17.399038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:17.398997    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" path="/var/lib/kubelet/pods/3f7c5551-d472-445c-b35f-b938c5dab57c/volumes"
Nov 02 14:52:17.438901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:17.438871    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"]
Nov 02 14:52:18.415054 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:18.415013    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:52:22.016444 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.016418    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:52:22.117574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.117540    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-token\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-token\") pod \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") "
Nov 02 14:52:22.117764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.117660    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-socket\") pod \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") "
Nov 02 14:52:22.117764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.117706    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-data\") pod \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") "
Nov 02 14:52:22.117971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.117845    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-envoy\") pod \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") "
Nov 02 14:52:22.117971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.117903    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-credential-socket\") pod \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") "
Nov 02 14:52:22.117971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.117944    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-podinfo\") pod \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") "
Nov 02 14:52:22.118107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.117997    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-kube-api-access-mllv6\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-kube-api-access-mllv6\") pod \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") "
Nov 02 14:52:22.118107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.118089    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-certs\") pod \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") "
Nov 02 14:52:22.118185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.118142    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istiod-ca-cert\") pod \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\" (UID: \"7650e8be-f936-48bf-9b6f-1d7674c46cbf\") "
Nov 02 14:52:22.118534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.118470    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-credential-socket" pod "7650e8be-f936-48bf-9b6f-1d7674c46cbf" (UID: "7650e8be-f936-48bf-9b6f-1d7674c46cbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:22.118652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.118569    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-data" pod "7650e8be-f936-48bf-9b6f-1d7674c46cbf" (UID: "7650e8be-f936-48bf-9b6f-1d7674c46cbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:22.118973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.118799    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-socket" pod "7650e8be-f936-48bf-9b6f-1d7674c46cbf" (UID: "7650e8be-f936-48bf-9b6f-1d7674c46cbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:22.118973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.118944    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.118973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.118967    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.119244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.118980    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.119244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.119209    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-certs" pod "7650e8be-f936-48bf-9b6f-1d7674c46cbf" (UID: "7650e8be-f936-48bf-9b6f-1d7674c46cbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:22.119535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.119511    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istiod-ca-cert" pod "7650e8be-f936-48bf-9b6f-1d7674c46cbf" (UID: "7650e8be-f936-48bf-9b6f-1d7674c46cbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:22.121427 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.121399    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-podinfo" pod "7650e8be-f936-48bf-9b6f-1d7674c46cbf" (UID: "7650e8be-f936-48bf-9b6f-1d7674c46cbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:52:22.121725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.121700    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-token" pod "7650e8be-f936-48bf-9b6f-1d7674c46cbf" (UID: "7650e8be-f936-48bf-9b6f-1d7674c46cbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:22.121952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.121930    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-envoy" pod "7650e8be-f936-48bf-9b6f-1d7674c46cbf" (UID: "7650e8be-f936-48bf-9b6f-1d7674c46cbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:22.123607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.123583    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-kube-api-access-mllv6" pod "7650e8be-f936-48bf-9b6f-1d7674c46cbf" (UID: "7650e8be-f936-48bf-9b6f-1d7674c46cbf"). InnerVolumeSpecName "kube-api-access-mllv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:22.220430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.220403    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.220430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.220427    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.220430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.220436    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mllv6\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-kube-api-access-mllv6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.220648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.220446    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7650e8be-f936-48bf-9b6f-1d7674c46cbf-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.220648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.220454    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.220648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.220462    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7650e8be-f936-48bf-9b6f-1d7674c46cbf-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.440141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.440101    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx" podUID="0fca14ad-ef87-4215-ae1d-2e760a55236b" containerName="payload-processing" containerID="cri-o://d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078" gracePeriod=25
Nov 02 14:52:22.502145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.502107    5034 generic.go:356] "Generic (PLEG): container finished" podID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerID="319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf" exitCode=0
Nov 02 14:52:22.502331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.502153    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" event={"ID":"7650e8be-f936-48bf-9b6f-1d7674c46cbf","Type":"ContainerDied","Data":"319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf"}
Nov 02 14:52:22.502331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.502206    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm" event={"ID":"7650e8be-f936-48bf-9b6f-1d7674c46cbf","Type":"ContainerDied","Data":"b768585907f261429bfa04743c55650b269f1e7f5db6bf55e7b627d12ece3561"}
Nov 02 14:52:22.502331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.502232    5034 scope.go:122] "RemoveContainer" containerID="319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf"
Nov 02 14:52:22.502331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.502164    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"
Nov 02 14:52:22.554937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.554913    5034 scope.go:122] "RemoveContainer" containerID="097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948"
Nov 02 14:52:22.562166 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.562142    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"]
Nov 02 14:52:22.566705 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.566684    5034 scope.go:122] "RemoveContainer" containerID="319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf"
Nov 02 14:52:22.567037 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:22.567017    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf\": container with ID starting with 319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf not found: ID does not exist" containerID="319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf"
Nov 02 14:52:22.567137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.567045    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf"} err="failed to get container status \"319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf\": rpc error: code = NotFound desc = could not find container \"319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf\": container with ID starting with 319bd135d67bf8be44cdc8a13cb802fd0e6fd94c59d89db5feb952a8676b9dbf not found: ID does not exist"
Nov 02 14:52:22.567137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.567068    5034 scope.go:122] "RemoveContainer" containerID="097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948"
Nov 02 14:52:22.567393 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:22.567370    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948\": container with ID starting with 097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948 not found: ID does not exist" containerID="097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948"
Nov 02 14:52:22.567513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.567396    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948"} err="failed to get container status \"097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948\": rpc error: code = NotFound desc = could not find container \"097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948\": container with ID starting with 097d8cf8cc3b0c0c11300e625257eddaec2bdabffc16dc09ce2ef562fea60948 not found: ID does not exist"
Nov 02 14:52:22.577589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.577563    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-bc34db-openshift-default-7769464b6f-zqhtm"]
Nov 02 14:52:22.757906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.757879    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:52:22.931305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.931270    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fca14ad-ef87-4215-ae1d-2e760a55236b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fca14ad-ef87-4215-ae1d-2e760a55236b-plugins-config-volume\") pod \"0fca14ad-ef87-4215-ae1d-2e760a55236b\" (UID: \"0fca14ad-ef87-4215-ae1d-2e760a55236b\") "
Nov 02 14:52:22.931498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.931381    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fca14ad-ef87-4215-ae1d-2e760a55236b-kube-api-access-762j4\" (UniqueName: \"kubernetes.io/projected/0fca14ad-ef87-4215-ae1d-2e760a55236b-kube-api-access-762j4\") pod \"0fca14ad-ef87-4215-ae1d-2e760a55236b\" (UID: \"0fca14ad-ef87-4215-ae1d-2e760a55236b\") "
Nov 02 14:52:22.931632 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.931608    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fca14ad-ef87-4215-ae1d-2e760a55236b-plugins-config-volume" pod "0fca14ad-ef87-4215-ae1d-2e760a55236b" (UID: "0fca14ad-ef87-4215-ae1d-2e760a55236b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:22.932012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.931990    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fca14ad-ef87-4215-ae1d-2e760a55236b-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:22.933530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:22.933507    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fca14ad-ef87-4215-ae1d-2e760a55236b-kube-api-access-762j4" pod "0fca14ad-ef87-4215-ae1d-2e760a55236b" (UID: "0fca14ad-ef87-4215-ae1d-2e760a55236b"). InnerVolumeSpecName "kube-api-access-762j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:23.033133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.033105    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-762j4\" (UniqueName: \"kubernetes.io/projected/0fca14ad-ef87-4215-ae1d-2e760a55236b-kube-api-access-762j4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:23.399651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.399571    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" path="/var/lib/kubelet/pods/7650e8be-f936-48bf-9b6f-1d7674c46cbf/volumes"
Nov 02 14:52:23.516345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.516312    5034 generic.go:356] "Generic (PLEG): container finished" podID="0fca14ad-ef87-4215-ae1d-2e760a55236b" containerID="d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078" exitCode=0
Nov 02 14:52:23.516622 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.516391    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"
Nov 02 14:52:23.517173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.516399    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx" event={"ID":"0fca14ad-ef87-4215-ae1d-2e760a55236b","Type":"ContainerDied","Data":"d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078"}
Nov 02 14:52:23.517173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.516846    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx" event={"ID":"0fca14ad-ef87-4215-ae1d-2e760a55236b","Type":"ContainerDied","Data":"fb5559c5dc4f3a79e9b76c93d622e5b5ece01b865266df7d10bc7160e8100b49"}
Nov 02 14:52:23.517173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.516889    5034 scope.go:122] "RemoveContainer" containerID="d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078"
Nov 02 14:52:23.535379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.535350    5034 scope.go:122] "RemoveContainer" containerID="d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078"
Nov 02 14:52:23.535748 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:23.535717    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078\": container with ID starting with d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078 not found: ID does not exist" containerID="d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078"
Nov 02 14:52:23.535872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.535757    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078"} err="failed to get container status \"d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078\": rpc error: code = NotFound desc = could not find container \"d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078\": container with ID starting with d13f1a06aee3b4627b67182b1f92f313340a695c84fa8c3cb6061e8d35922078 not found: ID does not exist"
Nov 02 14:52:23.578715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.576412    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"]
Nov 02 14:52:23.581412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.581377    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-605541-856fcdb955-jzwsx"]
Nov 02 14:52:23.927499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.927460    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"]
Nov 02 14:52:23.952829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:23.952799    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"]
Nov 02 14:52:25.397922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:25.397888    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fca14ad-ef87-4215-ae1d-2e760a55236b" path="/var/lib/kubelet/pods/0fca14ad-ef87-4215-ae1d-2e760a55236b/volumes"
Nov 02 14:52:28.928001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:28.927962    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z" podUID="f55f848e-281e-45a4-bb5f-77fea43cd292" containerName="payload-processing" containerID="cri-o://699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf" gracePeriod=25
Nov 02 14:52:28.954042 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:28.954011    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9" podUID="360e6c56-ada6-44b4-b491-dfb228dea4ad" containerName="payload-pre-processing" containerID="cri-o://c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562" gracePeriod=25
Nov 02 14:52:33.736019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:33.735986    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"]
Nov 02 14:52:33.736413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:33.736245    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy" containerID="cri-o://1ee3c666127e53a74aa9cf984536a6559e31ed1c571e9bbee2dc452f787283dd" gracePeriod=30
Nov 02 14:52:36.020760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:36.020724    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z" podUID="f55f848e-281e-45a4-bb5f-77fea43cd292" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:52:36.021207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:36.020851    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9" podUID="360e6c56-ada6-44b4-b491-dfb228dea4ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:52:39.042543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.042514    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:52:39.082156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082123    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-podinfo\") pod \"edbee88c-756f-40fe-a43f-3da83ea117a7\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") "
Nov 02 14:52:39.082303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082167    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-envoy\") pod \"edbee88c-756f-40fe-a43f-3da83ea117a7\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") "
Nov 02 14:52:39.082303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082203    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-data\") pod \"edbee88c-756f-40fe-a43f-3da83ea117a7\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") "
Nov 02 14:52:39.082303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082244    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-credential-socket\") pod \"edbee88c-756f-40fe-a43f-3da83ea117a7\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") "
Nov 02 14:52:39.082482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082340    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-certs\") pod \"edbee88c-756f-40fe-a43f-3da83ea117a7\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") "
Nov 02 14:52:39.082549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082482    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-token\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-token\") pod \"edbee88c-756f-40fe-a43f-3da83ea117a7\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") "
Nov 02 14:52:39.082549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082528    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-kube-api-access-lf6bk\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-kube-api-access-lf6bk\") pod \"edbee88c-756f-40fe-a43f-3da83ea117a7\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") "
Nov 02 14:52:39.082668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082569    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-data" pod "edbee88c-756f-40fe-a43f-3da83ea117a7" (UID: "edbee88c-756f-40fe-a43f-3da83ea117a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:39.082668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082603    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-credential-socket" pod "edbee88c-756f-40fe-a43f-3da83ea117a7" (UID: "edbee88c-756f-40fe-a43f-3da83ea117a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:39.082668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082619    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-certs" pod "edbee88c-756f-40fe-a43f-3da83ea117a7" (UID: "edbee88c-756f-40fe-a43f-3da83ea117a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:39.082668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082652    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edbee88c-756f-40fe-a43f-3da83ea117a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbee88c-756f-40fe-a43f-3da83ea117a7-istiod-ca-cert\") pod \"edbee88c-756f-40fe-a43f-3da83ea117a7\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") "
Nov 02 14:52:39.082950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.082717    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-socket\") pod \"edbee88c-756f-40fe-a43f-3da83ea117a7\" (UID: \"edbee88c-756f-40fe-a43f-3da83ea117a7\") "
Nov 02 14:52:39.083022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.083007    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-socket" pod "edbee88c-756f-40fe-a43f-3da83ea117a7" (UID: "edbee88c-756f-40fe-a43f-3da83ea117a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:39.083091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.083069    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edbee88c-756f-40fe-a43f-3da83ea117a7-istiod-ca-cert" pod "edbee88c-756f-40fe-a43f-3da83ea117a7" (UID: "edbee88c-756f-40fe-a43f-3da83ea117a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:39.083485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.083442    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:39.083485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.083474    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:39.083699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.083490    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edbee88c-756f-40fe-a43f-3da83ea117a7-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:39.083699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.083505    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:39.083699 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.083517    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:39.084909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.084882    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-podinfo" pod "edbee88c-756f-40fe-a43f-3da83ea117a7" (UID: "edbee88c-756f-40fe-a43f-3da83ea117a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:52:39.085014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.084943    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-envoy" pod "edbee88c-756f-40fe-a43f-3da83ea117a7" (UID: "edbee88c-756f-40fe-a43f-3da83ea117a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:52:39.085014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.084951    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-token" pod "edbee88c-756f-40fe-a43f-3da83ea117a7" (UID: "edbee88c-756f-40fe-a43f-3da83ea117a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:39.085101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.085047    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-kube-api-access-lf6bk" pod "edbee88c-756f-40fe-a43f-3da83ea117a7" (UID: "edbee88c-756f-40fe-a43f-3da83ea117a7"). InnerVolumeSpecName "kube-api-access-lf6bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:39.184845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.184815    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:39.184845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.184842    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf6bk\" (UniqueName: \"kubernetes.io/projected/edbee88c-756f-40fe-a43f-3da83ea117a7-kube-api-access-lf6bk\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:39.184845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.184852    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:39.185077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.184861    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edbee88c-756f-40fe-a43f-3da83ea117a7-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:39.310516 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.310442    5034 scope.go:122] "RemoveContainer" containerID="1b05e9c5cb8bd5b5bcdaf2bde4ea1efa31fc597dd62352341d1c4283103ab8d4"
Nov 02 14:52:39.323865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.323843    5034 scope.go:122] "RemoveContainer" containerID="2e4f925401a8fa0f3a41f51c03d79d7539dde864271740a4e48333eec76e6773"
Nov 02 14:52:39.343707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.343651    5034 scope.go:122] "RemoveContainer" containerID="1ee3c666127e53a74aa9cf984536a6559e31ed1c571e9bbee2dc452f787283dd"
Nov 02 14:52:39.709704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.709617    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"
Nov 02 14:52:39.709704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.709635    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" event={"ID":"edbee88c-756f-40fe-a43f-3da83ea117a7","Type":"ContainerDied","Data":"1ee3c666127e53a74aa9cf984536a6559e31ed1c571e9bbee2dc452f787283dd"}
Nov 02 14:52:39.709704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.709691    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq" event={"ID":"edbee88c-756f-40fe-a43f-3da83ea117a7","Type":"ContainerDied","Data":"d254ec49d99c9f36e92d6892bf5a45b6b396bda3a40ae5481657c812aa1b8576"}
Nov 02 14:52:39.799221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.799189    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"]
Nov 02 14:52:39.805192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.805162    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-605541-openshift-default-74db7c8949-9bnlq"]
Nov 02 14:52:39.912920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.912884    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"]
Nov 02 14:52:39.948971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:39.948943    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"]
Nov 02 14:52:41.397549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:41.397520    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" path="/var/lib/kubelet/pods/edbee88c-756f-40fe-a43f-3da83ea117a7/volumes"
Nov 02 14:52:42.037749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:42.037716    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"]
Nov 02 14:52:42.053128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:42.053092    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20d2338-v4kz7"]
Nov 02 14:52:43.397087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:43.397058    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa22c9ed-041e-4d67-af7b-4be66df234c5" path="/var/lib/kubelet/pods/fa22c9ed-041e-4d67-af7b-4be66df234c5/volumes"
Nov 02 14:52:44.913191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:44.913146    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn" podUID="e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d" containerName="payload-processing" containerID="cri-o://bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8" gracePeriod=25
Nov 02 14:52:44.949950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:44.949916    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp" podUID="06bc705b-a182-4a6f-b68c-5508e62e50b9" containerName="payload-pre-processing" containerID="cri-o://e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89" gracePeriod=25
Nov 02 14:52:45.283378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.283351    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:52:45.318745 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.318719    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:52:45.390176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.390148    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-plugins-config-volume\") pod \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\" (UID: \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\") "
Nov 02 14:52:45.390357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.390274    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-kube-api-access-jspbh\" (UniqueName: \"kubernetes.io/projected/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-kube-api-access-jspbh\") pod \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\" (UID: \"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d\") "
Nov 02 14:52:45.390912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.390888    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-plugins-config-volume" pod "e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d" (UID: "e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:45.393233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.393203    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-kube-api-access-jspbh" pod "e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d" (UID: "e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d"). InnerVolumeSpecName "kube-api-access-jspbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:45.491613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.491585    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bc705b-a182-4a6f-b68c-5508e62e50b9-kube-api-access-hpzfp\" (UniqueName: \"kubernetes.io/projected/06bc705b-a182-4a6f-b68c-5508e62e50b9-kube-api-access-hpzfp\") pod \"06bc705b-a182-4a6f-b68c-5508e62e50b9\" (UID: \"06bc705b-a182-4a6f-b68c-5508e62e50b9\") "
Nov 02 14:52:45.491751 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.491735    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06bc705b-a182-4a6f-b68c-5508e62e50b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bc705b-a182-4a6f-b68c-5508e62e50b9-plugins-config-volume\") pod \"06bc705b-a182-4a6f-b68c-5508e62e50b9\" (UID: \"06bc705b-a182-4a6f-b68c-5508e62e50b9\") "
Nov 02 14:52:45.492141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.492111    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bc705b-a182-4a6f-b68c-5508e62e50b9-plugins-config-volume" pod "06bc705b-a182-4a6f-b68c-5508e62e50b9" (UID: "06bc705b-a182-4a6f-b68c-5508e62e50b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:45.492276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.492246    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06bc705b-a182-4a6f-b68c-5508e62e50b9-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:45.492276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.492267    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspbh\" (UniqueName: \"kubernetes.io/projected/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-kube-api-access-jspbh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:45.492410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.492285    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:45.493837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.493815    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bc705b-a182-4a6f-b68c-5508e62e50b9-kube-api-access-hpzfp" pod "06bc705b-a182-4a6f-b68c-5508e62e50b9" (UID: "06bc705b-a182-4a6f-b68c-5508e62e50b9"). InnerVolumeSpecName "kube-api-access-hpzfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:45.593793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.593756    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpzfp\" (UniqueName: \"kubernetes.io/projected/06bc705b-a182-4a6f-b68c-5508e62e50b9-kube-api-access-hpzfp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:45.758753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.758658    5034 generic.go:356] "Generic (PLEG): container finished" podID="e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d" containerID="bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8" exitCode=0
Nov 02 14:52:45.758753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.758708    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"
Nov 02 14:52:45.758993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.758704    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn" event={"ID":"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d","Type":"ContainerDied","Data":"bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8"}
Nov 02 14:52:45.758993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.758841    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn" event={"ID":"e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d","Type":"ContainerDied","Data":"9638a3ca4a80461ca188e83bc520e88bb3472a03688a84c47efeb6b87554cc72"}
Nov 02 14:52:45.759500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.758874    5034 scope.go:122] "RemoveContainer" containerID="bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8"
Nov 02 14:52:45.776821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.775964    5034 generic.go:356] "Generic (PLEG): container finished" podID="06bc705b-a182-4a6f-b68c-5508e62e50b9" containerID="e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89" exitCode=0
Nov 02 14:52:45.776821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.776073    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"
Nov 02 14:52:45.776821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.776127    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp" event={"ID":"06bc705b-a182-4a6f-b68c-5508e62e50b9","Type":"ContainerDied","Data":"e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89"}
Nov 02 14:52:45.776821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.776215    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp" event={"ID":"06bc705b-a182-4a6f-b68c-5508e62e50b9","Type":"ContainerDied","Data":"368457bae2987d17c41da1c98aa0f84c7425cf41e0acc2dccfe4e9152ce387a5"}
Nov 02 14:52:45.786425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.786396    5034 scope.go:122] "RemoveContainer" containerID="bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8"
Nov 02 14:52:45.786747 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:45.786725    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8\": container with ID starting with bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8 not found: ID does not exist" containerID="bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8"
Nov 02 14:52:45.786866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.786757    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8"} err="failed to get container status \"bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8\": rpc error: code = NotFound desc = could not find container \"bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8\": container with ID starting with bc08587b3be5b3bfa930ba8ee7ef43eaa07ae3652aeb46802073521ff4ccdba8 not found: ID does not exist"
Nov 02 14:52:45.786866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.786814    5034 scope.go:122] "RemoveContainer" containerID="e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89"
Nov 02 14:52:45.798758 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.798736    5034 scope.go:122] "RemoveContainer" containerID="e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89"
Nov 02 14:52:45.799111 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:45.799086    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89\": container with ID starting with e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89 not found: ID does not exist" containerID="e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89"
Nov 02 14:52:45.799191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.799122    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89"} err="failed to get container status \"e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89\": rpc error: code = NotFound desc = could not find container \"e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89\": container with ID starting with e04a8b16cdcec1ecb6581ca8b400aafcddf630c57f079197c659aa4c8379ab89 not found: ID does not exist"
Nov 02 14:52:45.817336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.817314    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"]
Nov 02 14:52:45.831049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.831018    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-a4809a-7b485cdfcc-t9fnn"]
Nov 02 14:52:45.859121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.859100    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"]
Nov 02 14:52:45.863643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:45.863610    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a4809a-5c4bdb6775-lv4vp"]
Nov 02 14:52:46.020581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:46.020503    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z" podUID="f55f848e-281e-45a4-bb5f-77fea43cd292" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:52:46.020581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:46.020550    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9" podUID="360e6c56-ada6-44b4-b491-dfb228dea4ad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:52:47.400141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:47.400105    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bc705b-a182-4a6f-b68c-5508e62e50b9" path="/var/lib/kubelet/pods/06bc705b-a182-4a6f-b68c-5508e62e50b9/volumes"
Nov 02 14:52:47.400748 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:47.400727    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d" path="/var/lib/kubelet/pods/e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d/volumes"
Nov 02 14:52:48.415542 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:48.415503    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:52:49.264991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.264968    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:52:49.306051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.305873    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:52:49.354232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.353469    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/360e6c56-ada6-44b4-b491-dfb228dea4ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360e6c56-ada6-44b4-b491-dfb228dea4ad-plugins-config-volume\") pod \"360e6c56-ada6-44b4-b491-dfb228dea4ad\" (UID: \"360e6c56-ada6-44b4-b491-dfb228dea4ad\") "
Nov 02 14:52:49.354232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.353537    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/360e6c56-ada6-44b4-b491-dfb228dea4ad-kube-api-access-pll2m\" (UniqueName: \"kubernetes.io/projected/360e6c56-ada6-44b4-b491-dfb228dea4ad-kube-api-access-pll2m\") pod \"360e6c56-ada6-44b4-b491-dfb228dea4ad\" (UID: \"360e6c56-ada6-44b4-b491-dfb228dea4ad\") "
Nov 02 14:52:49.354232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.353577    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f55f848e-281e-45a4-bb5f-77fea43cd292-kube-api-access-dghx6\" (UniqueName: \"kubernetes.io/projected/f55f848e-281e-45a4-bb5f-77fea43cd292-kube-api-access-dghx6\") pod \"f55f848e-281e-45a4-bb5f-77fea43cd292\" (UID: \"f55f848e-281e-45a4-bb5f-77fea43cd292\") "
Nov 02 14:52:49.354232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.353634    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f55f848e-281e-45a4-bb5f-77fea43cd292-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f848e-281e-45a4-bb5f-77fea43cd292-plugins-config-volume\") pod \"f55f848e-281e-45a4-bb5f-77fea43cd292\" (UID: \"f55f848e-281e-45a4-bb5f-77fea43cd292\") "
Nov 02 14:52:49.354615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.354592    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f55f848e-281e-45a4-bb5f-77fea43cd292-plugins-config-volume" pod "f55f848e-281e-45a4-bb5f-77fea43cd292" (UID: "f55f848e-281e-45a4-bb5f-77fea43cd292"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:49.355082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.355058    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/360e6c56-ada6-44b4-b491-dfb228dea4ad-plugins-config-volume" pod "360e6c56-ada6-44b4-b491-dfb228dea4ad" (UID: "360e6c56-ada6-44b4-b491-dfb228dea4ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:52:49.358513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.357711    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/360e6c56-ada6-44b4-b491-dfb228dea4ad-kube-api-access-pll2m" pod "360e6c56-ada6-44b4-b491-dfb228dea4ad" (UID: "360e6c56-ada6-44b4-b491-dfb228dea4ad"). InnerVolumeSpecName "kube-api-access-pll2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:49.364831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.360485    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55f848e-281e-45a4-bb5f-77fea43cd292-kube-api-access-dghx6" pod "f55f848e-281e-45a4-bb5f-77fea43cd292" (UID: "f55f848e-281e-45a4-bb5f-77fea43cd292"). InnerVolumeSpecName "kube-api-access-dghx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:52:49.455646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.455510    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/360e6c56-ada6-44b4-b491-dfb228dea4ad-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:49.455646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.455541    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pll2m\" (UniqueName: \"kubernetes.io/projected/360e6c56-ada6-44b4-b491-dfb228dea4ad-kube-api-access-pll2m\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:49.455646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.455557    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dghx6\" (UniqueName: \"kubernetes.io/projected/f55f848e-281e-45a4-bb5f-77fea43cd292-kube-api-access-dghx6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:49.455646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.455572    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f55f848e-281e-45a4-bb5f-77fea43cd292-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:52:49.816029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.815990    5034 generic.go:356] "Generic (PLEG): container finished" podID="f55f848e-281e-45a4-bb5f-77fea43cd292" containerID="699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf" exitCode=0
Nov 02 14:52:49.816228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.816059    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"
Nov 02 14:52:49.816228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.816076    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z" event={"ID":"f55f848e-281e-45a4-bb5f-77fea43cd292","Type":"ContainerDied","Data":"699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf"}
Nov 02 14:52:49.816228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.816145    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z" event={"ID":"f55f848e-281e-45a4-bb5f-77fea43cd292","Type":"ContainerDied","Data":"6a116ed4e9fb53636d5d1bfde73b85534c0b41dee320a4bdfe0487eedada7bf7"}
Nov 02 14:52:49.816228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.816177    5034 scope.go:122] "RemoveContainer" containerID="699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf"
Nov 02 14:52:49.817730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.817702    5034 generic.go:356] "Generic (PLEG): container finished" podID="360e6c56-ada6-44b4-b491-dfb228dea4ad" containerID="c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562" exitCode=0
Nov 02 14:52:49.817900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.817754    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"
Nov 02 14:52:49.817900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.817754    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9" event={"ID":"360e6c56-ada6-44b4-b491-dfb228dea4ad","Type":"ContainerDied","Data":"c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562"}
Nov 02 14:52:49.817900 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.817826    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9" event={"ID":"360e6c56-ada6-44b4-b491-dfb228dea4ad","Type":"ContainerDied","Data":"cde78c2a03fefa5815e28707c6843d35b66f4b099c33d36523e02bc596d2a1e2"}
Nov 02 14:52:49.832147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.832105    5034 scope.go:122] "RemoveContainer" containerID="699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf"
Nov 02 14:52:49.832476 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:49.832449    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf\": container with ID starting with 699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf not found: ID does not exist" containerID="699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf"
Nov 02 14:52:49.832571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.832484    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf"} err="failed to get container status \"699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf\": rpc error: code = NotFound desc = could not find container \"699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf\": container with ID starting with 699a446c1e88ef89db7b70330912efd4f97031ac06d8cc8e81d928819ad10cbf not found: ID does not exist"
Nov 02 14:52:49.832571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.832506    5034 scope.go:122] "RemoveContainer" containerID="c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562"
Nov 02 14:52:49.842692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.842671    5034 scope.go:122] "RemoveContainer" containerID="c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562"
Nov 02 14:52:49.843010 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:52:49.842989    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562\": container with ID starting with c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562 not found: ID does not exist" containerID="c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562"
Nov 02 14:52:49.843105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.843018    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562"} err="failed to get container status \"c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562\": rpc error: code = NotFound desc = could not find container \"c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562\": container with ID starting with c5996130471dbc93575a8361b8c073aee82909bc7c81ada0f0e3a5ee2c4d8562 not found: ID does not exist"
Nov 02 14:52:49.875506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.875474    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"]
Nov 02 14:52:49.879278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.879254    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-4b7759-55d5fd9b8f-svj4z"]
Nov 02 14:52:49.916515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.916478    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"]
Nov 02 14:52:49.920518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:49.920496    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4b7759-798ffd8df9-n8wn9"]
Nov 02 14:52:51.397221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:51.397188    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="360e6c56-ada6-44b4-b491-dfb228dea4ad" path="/var/lib/kubelet/pods/360e6c56-ada6-44b4-b491-dfb228dea4ad/volumes"
Nov 02 14:52:51.397640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:51.397599    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f55f848e-281e-45a4-bb5f-77fea43cd292" path="/var/lib/kubelet/pods/f55f848e-281e-45a4-bb5f-77fea43cd292/volumes"
Nov 02 14:52:55.857175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:55.857133    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"]
Nov 02 14:52:55.857849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:55.857427    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy" containerID="cri-o://88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d" gracePeriod=30
Nov 02 14:52:59.030136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:59.030098    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"]
Nov 02 14:52:59.043143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:59.043116    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95607abd-8ng7r"]
Nov 02 14:52:59.396428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:52:59.396357    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61" path="/var/lib/kubelet/pods/98cb8f53-5a2b-4f50-bd49-8de4f0a4dd61/volumes"
Nov 02 14:53:00.480517 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.480485    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"]
Nov 02 14:53:00.481461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481436    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f28a0afb-2741-4a34-ab83-ff5af083c1a8" containerName="payload-pre-processing"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481462    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28a0afb-2741-4a34-ab83-ff5af083c1a8" containerName="payload-pre-processing"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481487    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3bf2886c-a17e-4255-a8be-5790ddacd3fd" containerName="revoke-keys"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481494    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf2886c-a17e-4255-a8be-5790ddacd3fd" containerName="revoke-keys"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481531    5034 cpu_manager.go:414] "containerMap: removing container" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481539    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481551    5034 cpu_manager.go:414] "containerMap: removing container" podUID="360e6c56-ada6-44b4-b491-dfb228dea4ad" containerName="payload-pre-processing"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481559    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="360e6c56-ada6-44b4-b491-dfb228dea4ad" containerName="payload-pre-processing"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481573    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481580    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy"
Nov 02 14:53:00.481596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481597    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d" containerName="payload-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481605    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d" containerName="payload-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481618    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f55f848e-281e-45a4-bb5f-77fea43cd292" containerName="payload-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481628    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f55f848e-281e-45a4-bb5f-77fea43cd292" containerName="payload-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481663    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481670    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481689    5034 cpu_manager.go:414] "containerMap: removing container" podUID="06bc705b-a182-4a6f-b68c-5508e62e50b9" containerName="payload-pre-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481696    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bc705b-a182-4a6f-b68c-5508e62e50b9" containerName="payload-pre-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481708    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481716    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481726    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0fca14ad-ef87-4215-ae1d-2e760a55236b" containerName="payload-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481734    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fca14ad-ef87-4215-ae1d-2e760a55236b" containerName="payload-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481750    5034 cpu_manager.go:414] "containerMap: removing container" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481757    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.481978    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482000    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bc705b-a182-4a6f-b68c-5508e62e50b9" containerName="payload-pre-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482012    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7c5551-d472-445c-b35f-b938c5dab57c" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482026    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fca14ad-ef87-4215-ae1d-2e760a55236b" containerName="payload-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482036    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482050    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482062    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28a0afb-2741-4a34-ab83-ff5af083c1a8" containerName="payload-pre-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482074    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf2886c-a17e-4255-a8be-5790ddacd3fd" containerName="revoke-keys"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482088    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbee88c-756f-40fe-a43f-3da83ea117a7" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482098    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="360e6c56-ada6-44b4-b491-dfb228dea4ad" containerName="payload-pre-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482110    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482123    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c0180a-6ee2-484d-99b2-0dd1ebfcd43d" containerName="payload-processing"
Nov 02 14:53:00.482214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.482131    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f55f848e-281e-45a4-bb5f-77fea43cd292" containerName="payload-processing"
Nov 02 14:53:00.487577 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.487554    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:00.504192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.504149    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:53:00.506109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.506078    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"]
Nov 02 14:53:00.516802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.514495    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:53:00.516802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.514525    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:53:00.516802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.514594    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:53:00.690224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.690183    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6m7\" (UniqueName: \"kubernetes.io/projected/ae536c7e-d84e-4412-bdca-9f96647be30f-kube-api-access-5l6m7\") pod \"maas-api-revoke-sub-b059ebab-kz6wb\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:00.690431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.690285    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b059ebab-kz6wb\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:00.690500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.690464    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b059ebab-kz6wb\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:00.814641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.799065    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b059ebab-kz6wb\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:00.814641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.799244    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6m7\" (UniqueName: \"kubernetes.io/projected/ae536c7e-d84e-4412-bdca-9f96647be30f-kube-api-access-5l6m7\") pod \"maas-api-revoke-sub-b059ebab-kz6wb\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:00.814641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.799320    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b059ebab-kz6wb\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:00.814641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.800195    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b059ebab-kz6wb\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:00.830800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.829263    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b059ebab-kz6wb\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:00.843414 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:00.843374    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6m7\" (UniqueName: \"kubernetes.io/projected/ae536c7e-d84e-4412-bdca-9f96647be30f-kube-api-access-5l6m7\") pod \"maas-api-revoke-sub-b059ebab-kz6wb\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:01.111232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.111201    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:01.322873 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.322113    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:53:01.416649 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.416619    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2c7637-7247-4cf6-a551-02f2fc51c21c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c7637-7247-4cf6-a551-02f2fc51c21c-istiod-ca-cert\") pod \"be2c7637-7247-4cf6-a551-02f2fc51c21c\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") "
Nov 02 14:53:01.416852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.416696    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-kube-api-access-2tn5b\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-kube-api-access-2tn5b\") pod \"be2c7637-7247-4cf6-a551-02f2fc51c21c\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") "
Nov 02 14:53:01.416852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.416796    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-credential-socket\") pod \"be2c7637-7247-4cf6-a551-02f2fc51c21c\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") "
Nov 02 14:53:01.416852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.416831    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-certs\") pod \"be2c7637-7247-4cf6-a551-02f2fc51c21c\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") "
Nov 02 14:53:01.417045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.416930    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-data\") pod \"be2c7637-7247-4cf6-a551-02f2fc51c21c\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") "
Nov 02 14:53:01.417045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.416976    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-podinfo\") pod \"be2c7637-7247-4cf6-a551-02f2fc51c21c\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") "
Nov 02 14:53:01.417045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.417017    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-token\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-token\") pod \"be2c7637-7247-4cf6-a551-02f2fc51c21c\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") "
Nov 02 14:53:01.417205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.417115    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-envoy\") pod \"be2c7637-7247-4cf6-a551-02f2fc51c21c\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") "
Nov 02 14:53:01.417205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.417179    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-socket\") pod \"be2c7637-7247-4cf6-a551-02f2fc51c21c\" (UID: \"be2c7637-7247-4cf6-a551-02f2fc51c21c\") "
Nov 02 14:53:01.418165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.417975    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-certs" pod "be2c7637-7247-4cf6-a551-02f2fc51c21c" (UID: "be2c7637-7247-4cf6-a551-02f2fc51c21c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:01.418465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.418387    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-data" pod "be2c7637-7247-4cf6-a551-02f2fc51c21c" (UID: "be2c7637-7247-4cf6-a551-02f2fc51c21c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:01.418465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.418422    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-socket" pod "be2c7637-7247-4cf6-a551-02f2fc51c21c" (UID: "be2c7637-7247-4cf6-a551-02f2fc51c21c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:01.418716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.418687    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-credential-socket" pod "be2c7637-7247-4cf6-a551-02f2fc51c21c" (UID: "be2c7637-7247-4cf6-a551-02f2fc51c21c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:01.419491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.419468    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2c7637-7247-4cf6-a551-02f2fc51c21c-istiod-ca-cert" pod "be2c7637-7247-4cf6-a551-02f2fc51c21c" (UID: "be2c7637-7247-4cf6-a551-02f2fc51c21c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:01.420628 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.420577    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-podinfo" pod "be2c7637-7247-4cf6-a551-02f2fc51c21c" (UID: "be2c7637-7247-4cf6-a551-02f2fc51c21c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:53:01.421703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.421657    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-kube-api-access-2tn5b" pod "be2c7637-7247-4cf6-a551-02f2fc51c21c" (UID: "be2c7637-7247-4cf6-a551-02f2fc51c21c"). InnerVolumeSpecName "kube-api-access-2tn5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:01.422060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.422040    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-envoy" pod "be2c7637-7247-4cf6-a551-02f2fc51c21c" (UID: "be2c7637-7247-4cf6-a551-02f2fc51c21c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:01.423410 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.423391    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-token" pod "be2c7637-7247-4cf6-a551-02f2fc51c21c" (UID: "be2c7637-7247-4cf6-a551-02f2fc51c21c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:01.469459 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:53:01.469428    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae536c7e_d84e_4412_bdca_9f96647be30f.slice/crio-9a089167a11121b79d46588ab87d76afd9fb858b37866043d5470657916b0017 WatchSource:0}: Error finding container 9a089167a11121b79d46588ab87d76afd9fb858b37866043d5470657916b0017: Status 404 returned error can't find the container with id 9a089167a11121b79d46588ab87d76afd9fb858b37866043d5470657916b0017
Nov 02 14:53:01.472440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.472408    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"]
Nov 02 14:53:01.519161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.519139    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.519161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.519162    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.519491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.519171    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.519491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.519179    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.519491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.519187    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.519491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.519194    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.519491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.519202    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2c7637-7247-4cf6-a551-02f2fc51c21c-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.519491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.519210    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2c7637-7247-4cf6-a551-02f2fc51c21c-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.519491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.519218    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tn5b\" (UniqueName: \"kubernetes.io/projected/be2c7637-7247-4cf6-a551-02f2fc51c21c-kube-api-access-2tn5b\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:01.929592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.929558    5034 generic.go:356] "Generic (PLEG): container finished" podID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerID="88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d" exitCode=0
Nov 02 14:53:01.929754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.929615    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"
Nov 02 14:53:01.929754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.929639    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" event={"ID":"be2c7637-7247-4cf6-a551-02f2fc51c21c","Type":"ContainerDied","Data":"88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d"}
Nov 02 14:53:01.929754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.929700    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h" event={"ID":"be2c7637-7247-4cf6-a551-02f2fc51c21c","Type":"ContainerDied","Data":"71a6a826f16e4cd03c55f37f5ddc2936b6f658ef8dca57008553775e37281f19"}
Nov 02 14:53:01.929754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.929728    5034 scope.go:122] "RemoveContainer" containerID="88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d"
Nov 02 14:53:01.931902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.931877    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb" event={"ID":"ae536c7e-d84e-4412-bdca-9f96647be30f","Type":"ContainerStarted","Data":"f3197027dce9208fa77e4441379580afe8cadb2d3b6be8da1debaabc5ee18ac3"}
Nov 02 14:53:01.932038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.931928    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb" event={"ID":"ae536c7e-d84e-4412-bdca-9f96647be30f","Type":"ContainerStarted","Data":"9a089167a11121b79d46588ab87d76afd9fb858b37866043d5470657916b0017"}
Nov 02 14:53:01.942254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.942235    5034 scope.go:122] "RemoveContainer" containerID="6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c"
Nov 02 14:53:01.953893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.953872    5034 scope.go:122] "RemoveContainer" containerID="88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d"
Nov 02 14:53:01.954196 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:53:01.954177    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d\": container with ID starting with 88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d not found: ID does not exist" containerID="88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d"
Nov 02 14:53:01.954303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.954204    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d"} err="failed to get container status \"88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d\": rpc error: code = NotFound desc = could not find container \"88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d\": container with ID starting with 88f102ab1d06014b244c0e8e34f5cf0f834efdb3d379244e6d8c5261b090597d not found: ID does not exist"
Nov 02 14:53:01.954303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.954225    5034 scope.go:122] "RemoveContainer" containerID="6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c"
Nov 02 14:53:01.954522 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:53:01.954504    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c\": container with ID starting with 6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c not found: ID does not exist" containerID="6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c"
Nov 02 14:53:01.954590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.954529    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c"} err="failed to get container status \"6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c\": rpc error: code = NotFound desc = could not find container \"6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c\": container with ID starting with 6394bc442ecbca4110fda64bba6db1f8337c9b6e83cd1995ef778de8c44dfe0c not found: ID does not exist"
Nov 02 14:53:01.990962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:01.990933    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"]
Nov 02 14:53:02.003429 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:02.003403    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e16d1c0e-openshift-default-544d8d9545-njn6h"]
Nov 02 14:53:02.033682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:02.033637    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb" podStartSLOduration=2.033624683 podStartE2EDuration="2.033624683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:02.031256621 +0000 UTC m=+4172.782561060" watchObservedRunningTime="2026-11-02 14:53:02.033624683 +0000 UTC m=+4172.784929120"
Nov 02 14:53:02.950194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:02.950158    5034 generic.go:356] "Generic (PLEG): container finished" podID="ae536c7e-d84e-4412-bdca-9f96647be30f" containerID="f3197027dce9208fa77e4441379580afe8cadb2d3b6be8da1debaabc5ee18ac3" exitCode=0
Nov 02 14:53:02.950685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:02.950232    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb" event={"ID":"ae536c7e-d84e-4412-bdca-9f96647be30f","Type":"ContainerDied","Data":"f3197027dce9208fa77e4441379580afe8cadb2d3b6be8da1debaabc5ee18ac3"}
Nov 02 14:53:03.395992 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:03.395961    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" path="/var/lib/kubelet/pods/be2c7637-7247-4cf6-a551-02f2fc51c21c/volumes"
Nov 02 14:53:04.139640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.139612    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:04.251426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.251396    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-xks-ca\") pod \"ae536c7e-d84e-4412-bdca-9f96647be30f\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") "
Nov 02 14:53:04.251640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.251522    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-openshift-ca\") pod \"ae536c7e-d84e-4412-bdca-9f96647be30f\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") "
Nov 02 14:53:04.251640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.251585    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae536c7e-d84e-4412-bdca-9f96647be30f-kube-api-access-5l6m7\" (UniqueName: \"kubernetes.io/projected/ae536c7e-d84e-4412-bdca-9f96647be30f-kube-api-access-5l6m7\") pod \"ae536c7e-d84e-4412-bdca-9f96647be30f\" (UID: \"ae536c7e-d84e-4412-bdca-9f96647be30f\") "
Nov 02 14:53:04.251997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.251956    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-openshift-ca" pod "ae536c7e-d84e-4412-bdca-9f96647be30f" (UID: "ae536c7e-d84e-4412-bdca-9f96647be30f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:04.254199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.254166    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-xks-ca" pod "ae536c7e-d84e-4412-bdca-9f96647be30f" (UID: "ae536c7e-d84e-4412-bdca-9f96647be30f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:04.256735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.256709    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae536c7e-d84e-4412-bdca-9f96647be30f-kube-api-access-5l6m7" pod "ae536c7e-d84e-4412-bdca-9f96647be30f" (UID: "ae536c7e-d84e-4412-bdca-9f96647be30f"). InnerVolumeSpecName "kube-api-access-5l6m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:04.353175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.353144    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:04.353175 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.353174    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l6m7\" (UniqueName: \"kubernetes.io/projected/ae536c7e-d84e-4412-bdca-9f96647be30f-kube-api-access-5l6m7\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:04.353404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.353188    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae536c7e-d84e-4412-bdca-9f96647be30f-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:04.967393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.967360    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"
Nov 02 14:53:04.967571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.967405    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb" event={"ID":"ae536c7e-d84e-4412-bdca-9f96647be30f","Type":"ContainerDied","Data":"9a089167a11121b79d46588ab87d76afd9fb858b37866043d5470657916b0017"}
Nov 02 14:53:04.967571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:04.967446    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a089167a11121b79d46588ab87d76afd9fb858b37866043d5470657916b0017"
Nov 02 14:53:05.619640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:05.619594    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"]
Nov 02 14:53:05.620204 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:05.619873    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy" containerID="cri-o://d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564" gracePeriod=30
Nov 02 14:53:10.901977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.901912    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"]
Nov 02 14:53:10.902875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.902852    5034 cpu_manager.go:414] "containerMap: removing container" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy"
Nov 02 14:53:10.902875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.902874    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy"
Nov 02 14:53:10.903057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.902895    5034 cpu_manager.go:414] "containerMap: removing container" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy"
Nov 02 14:53:10.903057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.902903    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy"
Nov 02 14:53:10.903057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.902926    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ae536c7e-d84e-4412-bdca-9f96647be30f" containerName="revoke-keys"
Nov 02 14:53:10.903057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.902934    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae536c7e-d84e-4412-bdca-9f96647be30f" containerName="revoke-keys"
Nov 02 14:53:10.903057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.902952    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy"
Nov 02 14:53:10.903057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.902959    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7650e8be-f936-48bf-9b6f-1d7674c46cbf" containerName="istio-proxy"
Nov 02 14:53:10.903371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.903189    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy"
Nov 02 14:53:10.903371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.903208    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2c7637-7247-4cf6-a551-02f2fc51c21c" containerName="istio-proxy"
Nov 02 14:53:10.903371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.903222    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae536c7e-d84e-4412-bdca-9f96647be30f" containerName="revoke-keys"
Nov 02 14:53:10.907069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.907046    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:10.916504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.916474    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:53:10.929093 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.929051    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"]
Nov 02 14:53:10.940484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.938149    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:53:10.940484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.938242    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:53:10.941196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.941170    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:53:10.941331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.941233    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:53:10.994981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.994944    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-certs\") pod \"f8ba3232-d954-4fa1-a47b-3453590c7922\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") "
Nov 02 14:53:10.994981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.994980    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-socket\") pod \"f8ba3232-d954-4fa1-a47b-3453590c7922\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") "
Nov 02 14:53:10.995249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995024    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8ba3232-d954-4fa1-a47b-3453590c7922-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ba3232-d954-4fa1-a47b-3453590c7922-istiod-ca-cert\") pod \"f8ba3232-d954-4fa1-a47b-3453590c7922\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") "
Nov 02 14:53:10.995249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995053    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-envoy\") pod \"f8ba3232-d954-4fa1-a47b-3453590c7922\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") "
Nov 02 14:53:10.995249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995074    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-token\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-token\") pod \"f8ba3232-d954-4fa1-a47b-3453590c7922\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") "
Nov 02 14:53:10.995249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995095    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-credential-socket\") pod \"f8ba3232-d954-4fa1-a47b-3453590c7922\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") "
Nov 02 14:53:10.995249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995149    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-data\") pod \"f8ba3232-d954-4fa1-a47b-3453590c7922\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") "
Nov 02 14:53:10.995249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995182    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-podinfo\") pod \"f8ba3232-d954-4fa1-a47b-3453590c7922\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") "
Nov 02 14:53:10.995249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995210    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-kube-api-access-hh7md\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-kube-api-access-hh7md\") pod \"f8ba3232-d954-4fa1-a47b-3453590c7922\" (UID: \"f8ba3232-d954-4fa1-a47b-3453590c7922\") "
Nov 02 14:53:10.995648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995481    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87fdb2d5-gf292\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:10.995648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995558    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87fdb2d5-gf292\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:10.995648 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.995622    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grnxt\" (UniqueName: \"kubernetes.io/projected/b4db072d-3be7-4a1a-9357-e06c49abc2c0-kube-api-access-grnxt\") pod \"maas-api-revoke-sub-87fdb2d5-gf292\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:10.996378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.996300    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-data" pod "f8ba3232-d954-4fa1-a47b-3453590c7922" (UID: "f8ba3232-d954-4fa1-a47b-3453590c7922"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:10.996378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.996330    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-certs" pod "f8ba3232-d954-4fa1-a47b-3453590c7922" (UID: "f8ba3232-d954-4fa1-a47b-3453590c7922"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:10.996378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.996311    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-socket" pod "f8ba3232-d954-4fa1-a47b-3453590c7922" (UID: "f8ba3232-d954-4fa1-a47b-3453590c7922"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:10.996688 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.996522    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-credential-socket" pod "f8ba3232-d954-4fa1-a47b-3453590c7922" (UID: "f8ba3232-d954-4fa1-a47b-3453590c7922"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:10.996773 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:10.996330    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8ba3232-d954-4fa1-a47b-3453590c7922-istiod-ca-cert" pod "f8ba3232-d954-4fa1-a47b-3453590c7922" (UID: "f8ba3232-d954-4fa1-a47b-3453590c7922"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:11.000040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.000015    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-token" pod "f8ba3232-d954-4fa1-a47b-3453590c7922" (UID: "f8ba3232-d954-4fa1-a47b-3453590c7922"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:11.000154 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.000062    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-kube-api-access-hh7md" pod "f8ba3232-d954-4fa1-a47b-3453590c7922" (UID: "f8ba3232-d954-4fa1-a47b-3453590c7922"). InnerVolumeSpecName "kube-api-access-hh7md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:11.007328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.004915    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-envoy" pod "f8ba3232-d954-4fa1-a47b-3453590c7922" (UID: "f8ba3232-d954-4fa1-a47b-3453590c7922"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:53:11.007328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.004986    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-podinfo" pod "f8ba3232-d954-4fa1-a47b-3453590c7922" (UID: "f8ba3232-d954-4fa1-a47b-3453590c7922"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:53:11.034877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.034285    5034 generic.go:356] "Generic (PLEG): container finished" podID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerID="d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564" exitCode=0
Nov 02 14:53:11.034877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.034358    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"
Nov 02 14:53:11.034877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.034383    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" event={"ID":"f8ba3232-d954-4fa1-a47b-3453590c7922","Type":"ContainerDied","Data":"d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564"}
Nov 02 14:53:11.034877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.034444    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5" event={"ID":"f8ba3232-d954-4fa1-a47b-3453590c7922","Type":"ContainerDied","Data":"00c67923f0f866f63a84eb091ffb81b2fd89aa805797862466a02b26049cbe5e"}
Nov 02 14:53:11.034877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.034460    5034 scope.go:122] "RemoveContainer" containerID="d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564"
Nov 02 14:53:11.034877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.034471    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"]
Nov 02 14:53:11.038083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.038059    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691c3c9a-9j27q"]
Nov 02 14:53:11.052536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.052508    5034 scope.go:122] "RemoveContainer" containerID="8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49"
Nov 02 14:53:11.064451 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.064436    5034 scope.go:122] "RemoveContainer" containerID="d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564"
Nov 02 14:53:11.064711 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:53:11.064693    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564\": container with ID starting with d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564 not found: ID does not exist" containerID="d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564"
Nov 02 14:53:11.064762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.064719    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564"} err="failed to get container status \"d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564\": rpc error: code = NotFound desc = could not find container \"d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564\": container with ID starting with d1dd37ac2b8cb115efd48e9e836a823dcf3d9e7e812815324639f63a46daa564 not found: ID does not exist"
Nov 02 14:53:11.064762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.064734    5034 scope.go:122] "RemoveContainer" containerID="8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49"
Nov 02 14:53:11.065022 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:53:11.065005    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49\": container with ID starting with 8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49 not found: ID does not exist" containerID="8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49"
Nov 02 14:53:11.065074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.065027    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49"} err="failed to get container status \"8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49\": rpc error: code = NotFound desc = could not find container \"8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49\": container with ID starting with 8a831f22ea4ea1cee1d3806b3d473c82a0007c9caf5044d658599334f08a6a49 not found: ID does not exist"
Nov 02 14:53:11.096963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.096933    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87fdb2d5-gf292\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:11.097114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097046    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87fdb2d5-gf292\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:11.097193 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097135    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grnxt\" (UniqueName: \"kubernetes.io/projected/b4db072d-3be7-4a1a-9357-e06c49abc2c0-kube-api-access-grnxt\") pod \"maas-api-revoke-sub-87fdb2d5-gf292\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:11.097412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097389    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:11.097513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097414    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:11.097513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097428    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8ba3232-d954-4fa1-a47b-3453590c7922-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:11.097513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097441    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:11.097513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097453    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:11.097513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097465    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:11.097513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097480    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:11.097513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097493    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8ba3232-d954-4fa1-a47b-3453590c7922-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:11.097513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097506    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh7md\" (UniqueName: \"kubernetes.io/projected/f8ba3232-d954-4fa1-a47b-3453590c7922-kube-api-access-hh7md\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:11.097961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.097850    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87fdb2d5-gf292\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:11.099708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.099683    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87fdb2d5-gf292\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:11.101387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.101366    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"]
Nov 02 14:53:11.107043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.107013    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a4809a-openshift-default-68c49dcdc5-lhbn5"]
Nov 02 14:53:11.122556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.122533    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grnxt\" (UniqueName: \"kubernetes.io/projected/b4db072d-3be7-4a1a-9357-e06c49abc2c0-kube-api-access-grnxt\") pod \"maas-api-revoke-sub-87fdb2d5-gf292\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:11.253678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.253648    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:11.397542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.397505    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2bf2265-7b80-4403-8844-71b555a59bcc" path="/var/lib/kubelet/pods/a2bf2265-7b80-4403-8844-71b555a59bcc/volumes"
Nov 02 14:53:11.398214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.398188    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" path="/var/lib/kubelet/pods/f8ba3232-d954-4fa1-a47b-3453590c7922/volumes"
Nov 02 14:53:11.568319 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:53:11.568290    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4db072d_3be7_4a1a_9357_e06c49abc2c0.slice/crio-c52a7cc6b30257bef46920069d94340b6129c9e862d26e0f33b2d9ffabbd1d08 WatchSource:0}: Error finding container c52a7cc6b30257bef46920069d94340b6129c9e862d26e0f33b2d9ffabbd1d08: Status 404 returned error can't find the container with id c52a7cc6b30257bef46920069d94340b6129c9e862d26e0f33b2d9ffabbd1d08
Nov 02 14:53:11.569109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.569081    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"]
Nov 02 14:53:11.975942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.975909    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"]
Nov 02 14:53:11.976659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.976642    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy"
Nov 02 14:53:11.976659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.976658    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy"
Nov 02 14:53:11.976812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.976668    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy"
Nov 02 14:53:11.976812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.976673    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy"
Nov 02 14:53:11.977031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.976854    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy"
Nov 02 14:53:11.977031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.976866    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8ba3232-d954-4fa1-a47b-3453590c7922" containerName="istio-proxy"
Nov 02 14:53:11.980865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:11.980844    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.007202 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.007174    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-960a68-openshift-default-dockercfg-b6hdg\""
Nov 02 14:53:12.007332 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.007267    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"]
Nov 02 14:53:12.050803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.050740    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292" event={"ID":"b4db072d-3be7-4a1a-9357-e06c49abc2c0","Type":"ContainerStarted","Data":"14e50cc250000e7dc6cd97023bf062acf273461119e64c881b7c283a94a22d88"}
Nov 02 14:53:12.050964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.050825    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292" event={"ID":"b4db072d-3be7-4a1a-9357-e06c49abc2c0","Type":"ContainerStarted","Data":"c52a7cc6b30257bef46920069d94340b6129c9e862d26e0f33b2d9ffabbd1d08"}
Nov 02 14:53:12.098575 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.098522    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292" podStartSLOduration=2.098509415 podStartE2EDuration="2.098509415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:12.093699707 +0000 UTC m=+4182.845004145" watchObservedRunningTime="2026-11-02 14:53:12.098509415 +0000 UTC m=+4182.849813850"
Nov 02 14:53:12.112868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.112831    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4p5\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-kube-api-access-jz4p5\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.113019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.112966    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-certs\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.113019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.112995    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cab00b6-3287-4f34-a7e3-23fc811d615c-istiod-ca-cert\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.113019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.113017    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-data\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.113196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.113059    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.113255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.113216    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-token\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.113255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.113241    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-credential-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.113371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.113287    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-podinfo\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.113425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.113393    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-envoy\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.214385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.214346    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4p5\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-kube-api-access-jz4p5\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.214552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.214445    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-certs\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.214552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.214477    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cab00b6-3287-4f34-a7e3-23fc811d615c-istiod-ca-cert\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.214762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.214621    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-data\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.214762 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.214699    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.214933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.214846    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-token\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.214933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.214882    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-credential-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.215067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.214957    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-podinfo\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.215067 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.214980    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-certs\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.215187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.215138    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-envoy\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.215187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.215148    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-data\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.215187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.215171    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.215375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.215191    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-credential-socket\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.215375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.215326    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cab00b6-3287-4f34-a7e3-23fc811d615c-istiod-ca-cert\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.217584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.217562    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-envoy\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.217851 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.217829    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-podinfo\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.239559 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.239533    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4p5\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-kube-api-access-jz4p5\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.239932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.239914    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-token\") pod \"e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") " pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.297979 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.297951    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:12.636619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.636581    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"]
Nov 02 14:53:12.644056 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:53:12.644031    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cab00b6_3287_4f34_a7e3_23fc811d615c.slice/crio-8e2b190321b307dfb16c201b079bd5d3c69acf3fbf05765ba2f05d9de12bd65d WatchSource:0}: Error finding container 8e2b190321b307dfb16c201b079bd5d3c69acf3fbf05765ba2f05d9de12bd65d: Status 404 returned error can't find the container with id 8e2b190321b307dfb16c201b079bd5d3c69acf3fbf05765ba2f05d9de12bd65d
Nov 02 14:53:12.647018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.646984    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:53:12.647125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.647063    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:53:12.647125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.647096    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:53:12.838035 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:12.837945    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"]
Nov 02 14:53:13.077678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:13.077635    5034 generic.go:356] "Generic (PLEG): container finished" podID="b4db072d-3be7-4a1a-9357-e06c49abc2c0" containerID="14e50cc250000e7dc6cd97023bf062acf273461119e64c881b7c283a94a22d88" exitCode=0
Nov 02 14:53:13.078153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:13.077855    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292" event={"ID":"b4db072d-3be7-4a1a-9357-e06c49abc2c0","Type":"ContainerDied","Data":"14e50cc250000e7dc6cd97023bf062acf273461119e64c881b7c283a94a22d88"}
Nov 02 14:53:13.080619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:13.080591    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" event={"ID":"5cab00b6-3287-4f34-a7e3-23fc811d615c","Type":"ContainerStarted","Data":"38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973"}
Nov 02 14:53:13.080834 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:13.080644    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" event={"ID":"5cab00b6-3287-4f34-a7e3-23fc811d615c","Type":"ContainerStarted","Data":"8e2b190321b307dfb16c201b079bd5d3c69acf3fbf05765ba2f05d9de12bd65d"}
Nov 02 14:53:13.178654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:13.178542    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" podStartSLOduration=2.178524675 podStartE2EDuration="2.178524675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:13.173766985 +0000 UTC m=+4183.925071416" watchObservedRunningTime="2026-11-02 14:53:13.178524675 +0000 UTC m=+4183.929829116"
Nov 02 14:53:13.298184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:13.298146    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:13.299745 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:13.299715    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body=
Nov 02 14:53:13.299923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:13.299763    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused"
Nov 02 14:53:14.301801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.301263    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body=
Nov 02 14:53:14.301801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.301344    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused"
Nov 02 14:53:14.450932 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.450906    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:14.544536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.544502    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4db072d-3be7-4a1a-9357-e06c49abc2c0-kube-api-access-grnxt\" (UniqueName: \"kubernetes.io/projected/b4db072d-3be7-4a1a-9357-e06c49abc2c0-kube-api-access-grnxt\") pod \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") "
Nov 02 14:53:14.544716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.544630    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-openshift-ca\") pod \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") "
Nov 02 14:53:14.544716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.544696    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-xks-ca\") pod \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\" (UID: \"b4db072d-3be7-4a1a-9357-e06c49abc2c0\") "
Nov 02 14:53:14.545472 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.545445    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-openshift-ca" pod "b4db072d-3be7-4a1a-9357-e06c49abc2c0" (UID: "b4db072d-3be7-4a1a-9357-e06c49abc2c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:14.546942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.546914    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-xks-ca" pod "b4db072d-3be7-4a1a-9357-e06c49abc2c0" (UID: "b4db072d-3be7-4a1a-9357-e06c49abc2c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:14.547249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.547227    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4db072d-3be7-4a1a-9357-e06c49abc2c0-kube-api-access-grnxt" pod "b4db072d-3be7-4a1a-9357-e06c49abc2c0" (UID: "b4db072d-3be7-4a1a-9357-e06c49abc2c0"). InnerVolumeSpecName "kube-api-access-grnxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:14.646375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.646333    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:14.646375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.646368    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4db072d-3be7-4a1a-9357-e06c49abc2c0-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:14.646375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:14.646382    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grnxt\" (UniqueName: \"kubernetes.io/projected/b4db072d-3be7-4a1a-9357-e06c49abc2c0-kube-api-access-grnxt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:15.138705 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:15.138677    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"
Nov 02 14:53:15.138912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:15.138708    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292" event={"ID":"b4db072d-3be7-4a1a-9357-e06c49abc2c0","Type":"ContainerDied","Data":"c52a7cc6b30257bef46920069d94340b6129c9e862d26e0f33b2d9ffabbd1d08"}
Nov 02 14:53:15.138912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:15.138766    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c52a7cc6b30257bef46920069d94340b6129c9e862d26e0f33b2d9ffabbd1d08"
Nov 02 14:53:15.300861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:15.299140    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body=
Nov 02 14:53:15.300861 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:15.299211    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused"
Nov 02 14:53:15.527250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:15.527215    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"]
Nov 02 14:53:16.300275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:16.300234    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body=
Nov 02 14:53:16.300570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:16.300541    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused"
Nov 02 14:53:17.301077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:17.301034    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body=
Nov 02 14:53:17.301486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:17.301101    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused"
Nov 02 14:53:17.840803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:17.838921    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95" podUID="95588e4f-c3d8-4cd7-acea-b4efb1b457c7" containerName="payload-pre-processing" containerID="cri-o://8119ba0643256a9041a041550bcf0cbbf5aae3040a880ac507c7ce0d1cd62d8c" gracePeriod=25
Nov 02 14:53:18.221044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.220950    5034 generic.go:356] "Generic (PLEG): container finished" podID="95588e4f-c3d8-4cd7-acea-b4efb1b457c7" containerID="8119ba0643256a9041a041550bcf0cbbf5aae3040a880ac507c7ce0d1cd62d8c" exitCode=0
Nov 02 14:53:18.221278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.221251    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95" event={"ID":"95588e4f-c3d8-4cd7-acea-b4efb1b457c7","Type":"ContainerDied","Data":"8119ba0643256a9041a041550bcf0cbbf5aae3040a880ac507c7ce0d1cd62d8c"}
Nov 02 14:53:18.299258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.299222    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body=
Nov 02 14:53:18.299515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.299487    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused"
Nov 02 14:53:18.458665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.458630    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:53:18.610159 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:53:18.600679    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:53:18.628816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.627078    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-kube-api-access-vfjmj\" (UniqueName: \"kubernetes.io/projected/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-kube-api-access-vfjmj\") pod \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\" (UID: \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\") "
Nov 02 14:53:18.628816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.627199    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-plugins-config-volume\") pod \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\" (UID: \"95588e4f-c3d8-4cd7-acea-b4efb1b457c7\") "
Nov 02 14:53:18.646852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.644074    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-plugins-config-volume" pod "95588e4f-c3d8-4cd7-acea-b4efb1b457c7" (UID: "95588e4f-c3d8-4cd7-acea-b4efb1b457c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:18.723482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.723435    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-kube-api-access-vfjmj" pod "95588e4f-c3d8-4cd7-acea-b4efb1b457c7" (UID: "95588e4f-c3d8-4cd7-acea-b4efb1b457c7"). InnerVolumeSpecName "kube-api-access-vfjmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:18.734991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.733931    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:18.734991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:18.733968    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfjmj\" (UniqueName: \"kubernetes.io/projected/95588e4f-c3d8-4cd7-acea-b4efb1b457c7-kube-api-access-vfjmj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:19.233426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:19.233390    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"
Nov 02 14:53:19.233426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:19.233406    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95" event={"ID":"95588e4f-c3d8-4cd7-acea-b4efb1b457c7","Type":"ContainerDied","Data":"fc8b49a9c7d1fa8447d815f6487dd933e8c7c889aa15779c412c8a3f6c41b224"}
Nov 02 14:53:19.233636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:19.233507    5034 scope.go:122] "RemoveContainer" containerID="8119ba0643256a9041a041550bcf0cbbf5aae3040a880ac507c7ce0d1cd62d8c"
Nov 02 14:53:19.298868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:19.298451    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused" start-of-body=
Nov 02 14:53:19.298868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:19.298510    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.155:15021/healthz/ready\": dial tcp 10.132.1.155:15021: connect: connection refused"
Nov 02 14:53:19.300725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:19.300688    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"]
Nov 02 14:53:19.305795 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:19.305748    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-662e3e-954f47679-mdl95"]
Nov 02 14:53:19.398489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:19.398457    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95588e4f-c3d8-4cd7-acea-b4efb1b457c7" path="/var/lib/kubelet/pods/95588e4f-c3d8-4cd7-acea-b4efb1b457c7/volumes"
Nov 02 14:53:20.304796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:20.304750    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:20.305317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:20.305291    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:20.306123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:20.306101    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:53:20.528257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:20.528207    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd" podUID="5e620544-fbdd-41c4-baf3-7bfc9aaf13ba" containerName="payload-pre-processing" containerID="cri-o://9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc" gracePeriod=25
Nov 02 14:53:20.818530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:20.818504    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:53:20.967695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:20.967621    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-kube-api-access-6nnjc\" (UniqueName: \"kubernetes.io/projected/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-kube-api-access-6nnjc\") pod \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\" (UID: \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\") "
Nov 02 14:53:20.967879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:20.967842    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-plugins-config-volume\") pod \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\" (UID: \"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba\") "
Nov 02 14:53:20.968195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:20.968171    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-plugins-config-volume" pod "5e620544-fbdd-41c4-baf3-7bfc9aaf13ba" (UID: "5e620544-fbdd-41c4-baf3-7bfc9aaf13ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:20.969848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:20.969822    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-kube-api-access-6nnjc" pod "5e620544-fbdd-41c4-baf3-7bfc9aaf13ba" (UID: "5e620544-fbdd-41c4-baf3-7bfc9aaf13ba"). InnerVolumeSpecName "kube-api-access-6nnjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:21.069671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.069639    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:21.069671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.069670    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nnjc\" (UniqueName: \"kubernetes.io/projected/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba-kube-api-access-6nnjc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:21.254354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.254320    5034 generic.go:356] "Generic (PLEG): container finished" podID="5e620544-fbdd-41c4-baf3-7bfc9aaf13ba" containerID="9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc" exitCode=0
Nov 02 14:53:21.254532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.254358    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"
Nov 02 14:53:21.254532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.254403    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd" event={"ID":"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba","Type":"ContainerDied","Data":"9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc"}
Nov 02 14:53:21.254532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.254462    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd" event={"ID":"5e620544-fbdd-41c4-baf3-7bfc9aaf13ba","Type":"ContainerDied","Data":"f0d9d1ba1eb2432a74ddafd4e8ceb9a862bec18066c147ab277cadd7b87c70f4"}
Nov 02 14:53:21.254532 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.254486    5034 scope.go:122] "RemoveContainer" containerID="9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc"
Nov 02 14:53:21.266159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.266139    5034 scope.go:122] "RemoveContainer" containerID="9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc"
Nov 02 14:53:21.266477 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:53:21.266443    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc\": container with ID starting with 9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc not found: ID does not exist" containerID="9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc"
Nov 02 14:53:21.266585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.266483    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc"} err="failed to get container status \"9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc\": rpc error: code = NotFound desc = could not find container \"9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc\": container with ID starting with 9454c94b92c3ab37a2503949f1300a5c7337deacdb8842b0ab94f1d6a25175cc not found: ID does not exist"
Nov 02 14:53:21.314152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.314120    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"]
Nov 02 14:53:21.319440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.319416    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-2daae4-646c5fd5c5-mrggd"]
Nov 02 14:53:21.397161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.397133    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e620544-fbdd-41c4-baf3-7bfc9aaf13ba" path="/var/lib/kubelet/pods/5e620544-fbdd-41c4-baf3-7bfc9aaf13ba/volumes"
Nov 02 14:53:21.773108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.773070    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"]
Nov 02 14:53:21.773882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.773860    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5e620544-fbdd-41c4-baf3-7bfc9aaf13ba" containerName="payload-pre-processing"
Nov 02 14:53:21.773882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.773882    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e620544-fbdd-41c4-baf3-7bfc9aaf13ba" containerName="payload-pre-processing"
Nov 02 14:53:21.774070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.773919    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b4db072d-3be7-4a1a-9357-e06c49abc2c0" containerName="revoke-keys"
Nov 02 14:53:21.774070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.773927    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4db072d-3be7-4a1a-9357-e06c49abc2c0" containerName="revoke-keys"
Nov 02 14:53:21.774070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.773957    5034 cpu_manager.go:414] "containerMap: removing container" podUID="95588e4f-c3d8-4cd7-acea-b4efb1b457c7" containerName="payload-pre-processing"
Nov 02 14:53:21.774070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.773964    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95588e4f-c3d8-4cd7-acea-b4efb1b457c7" containerName="payload-pre-processing"
Nov 02 14:53:21.774283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.774134    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e620544-fbdd-41c4-baf3-7bfc9aaf13ba" containerName="payload-pre-processing"
Nov 02 14:53:21.774283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.774157    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4db072d-3be7-4a1a-9357-e06c49abc2c0" containerName="revoke-keys"
Nov 02 14:53:21.774283 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.774174    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="95588e4f-c3d8-4cd7-acea-b4efb1b457c7" containerName="payload-pre-processing"
Nov 02 14:53:21.779300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.779279    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:21.796950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.796929    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:53:21.797098 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.796933    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:53:21.797201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.797102    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:53:21.797291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.797269    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:53:21.798518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.798492    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"]
Nov 02 14:53:21.879485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.879455    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwbl\" (UniqueName: \"kubernetes.io/projected/cb73b587-e811-412c-aa3a-7c16ded971db-kube-api-access-8mwbl\") pod \"maas-api-revoke-sub-c4a38947-mrvnq\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:21.879695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.879530    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a38947-mrvnq\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:21.879764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.879724    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a38947-mrvnq\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:21.980605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.980567    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a38947-mrvnq\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:21.980820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.980688    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwbl\" (UniqueName: \"kubernetes.io/projected/cb73b587-e811-412c-aa3a-7c16ded971db-kube-api-access-8mwbl\") pod \"maas-api-revoke-sub-c4a38947-mrvnq\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:21.980820 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.980767    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a38947-mrvnq\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:21.981562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.981534    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a38947-mrvnq\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:21.983103 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:21.983076    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a38947-mrvnq\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:22.005234 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:22.005209    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwbl\" (UniqueName: \"kubernetes.io/projected/cb73b587-e811-412c-aa3a-7c16ded971db-kube-api-access-8mwbl\") pod \"maas-api-revoke-sub-c4a38947-mrvnq\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:22.099455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:22.099396    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:22.415713 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:53:22.415682    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb73b587_e811_412c_aa3a_7c16ded971db.slice/crio-68b693ae945c12f4e917ec6fb9f886fa264ca358e242eb77e619d99677cf61ed WatchSource:0}: Error finding container 68b693ae945c12f4e917ec6fb9f886fa264ca358e242eb77e619d99677cf61ed: Status 404 returned error can't find the container with id 68b693ae945c12f4e917ec6fb9f886fa264ca358e242eb77e619d99677cf61ed
Nov 02 14:53:22.416918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:22.416889    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"]
Nov 02 14:53:23.273542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:23.273507    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq" event={"ID":"cb73b587-e811-412c-aa3a-7c16ded971db","Type":"ContainerStarted","Data":"57ae871587bb11afa5b67ada3c437552e33cebcd01638f85b605b52ce01a1bfb"}
Nov 02 14:53:23.273735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:23.273561    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq" event={"ID":"cb73b587-e811-412c-aa3a-7c16ded971db","Type":"ContainerStarted","Data":"68b693ae945c12f4e917ec6fb9f886fa264ca358e242eb77e619d99677cf61ed"}
Nov 02 14:53:23.323596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:23.323545    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq" podStartSLOduration=2.323530537 podStartE2EDuration="2.323530537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:23.323065877 +0000 UTC m=+4194.074370312" watchObservedRunningTime="2026-11-02 14:53:23.323530537 +0000 UTC m=+4194.074834974"
Nov 02 14:53:24.289944 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:24.289910    5034 generic.go:356] "Generic (PLEG): container finished" podID="cb73b587-e811-412c-aa3a-7c16ded971db" containerID="57ae871587bb11afa5b67ada3c437552e33cebcd01638f85b605b52ce01a1bfb" exitCode=0
Nov 02 14:53:24.290394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:24.289989    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq" event={"ID":"cb73b587-e811-412c-aa3a-7c16ded971db","Type":"ContainerDied","Data":"57ae871587bb11afa5b67ada3c437552e33cebcd01638f85b605b52ce01a1bfb"}
Nov 02 14:53:25.464300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.464271    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:25.540677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.540646    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-openshift-ca\") pod \"cb73b587-e811-412c-aa3a-7c16ded971db\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") "
Nov 02 14:53:25.540832 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.540817    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-xks-ca\") pod \"cb73b587-e811-412c-aa3a-7c16ded971db\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") "
Nov 02 14:53:25.540942 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.540924    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb73b587-e811-412c-aa3a-7c16ded971db-kube-api-access-8mwbl\" (UniqueName: \"kubernetes.io/projected/cb73b587-e811-412c-aa3a-7c16ded971db-kube-api-access-8mwbl\") pod \"cb73b587-e811-412c-aa3a-7c16ded971db\" (UID: \"cb73b587-e811-412c-aa3a-7c16ded971db\") "
Nov 02 14:53:25.541290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.541076    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-openshift-ca" pod "cb73b587-e811-412c-aa3a-7c16ded971db" (UID: "cb73b587-e811-412c-aa3a-7c16ded971db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:25.541528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.541503    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:25.543251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.543219    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb73b587-e811-412c-aa3a-7c16ded971db-kube-api-access-8mwbl" pod "cb73b587-e811-412c-aa3a-7c16ded971db" (UID: "cb73b587-e811-412c-aa3a-7c16ded971db"). InnerVolumeSpecName "kube-api-access-8mwbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:25.543509 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.543480    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-xks-ca" pod "cb73b587-e811-412c-aa3a-7c16ded971db" (UID: "cb73b587-e811-412c-aa3a-7c16ded971db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:25.643162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.643136    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb73b587-e811-412c-aa3a-7c16ded971db-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:25.643162 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:25.643159    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mwbl\" (UniqueName: \"kubernetes.io/projected/cb73b587-e811-412c-aa3a-7c16ded971db-kube-api-access-8mwbl\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:26.307665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:26.307634    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"
Nov 02 14:53:26.307903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:26.307660    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq" event={"ID":"cb73b587-e811-412c-aa3a-7c16ded971db","Type":"ContainerDied","Data":"68b693ae945c12f4e917ec6fb9f886fa264ca358e242eb77e619d99677cf61ed"}
Nov 02 14:53:26.307903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:26.307716    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68b693ae945c12f4e917ec6fb9f886fa264ca358e242eb77e619d99677cf61ed"
Nov 02 14:53:27.335530 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.335500    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"]
Nov 02 14:53:27.336535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.336510    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cb73b587-e811-412c-aa3a-7c16ded971db" containerName="revoke-keys"
Nov 02 14:53:27.336535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.336532    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb73b587-e811-412c-aa3a-7c16ded971db" containerName="revoke-keys"
Nov 02 14:53:27.341018 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.340982    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb73b587-e811-412c-aa3a-7c16ded971db" containerName="revoke-keys"
Nov 02 14:53:27.348665 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.348642    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.367169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.367140    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"]
Nov 02 14:53:27.377703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.377675    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fcefcc-openshift-default-dockercfg-r7kkx\""
Nov 02 14:53:27.468540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.468511    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw6gs\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-kube-api-access-lw6gs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.468691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.468548    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.468691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.468585    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.468691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.468618    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.468878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.468714    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.468878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.468805    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.468990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.468930    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.469048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.468990    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.469048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.469018    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570208    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570254    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570338    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6gs\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-kube-api-access-lw6gs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570365    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570407    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570431    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570574    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570682 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570672    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570768    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.570938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.570882    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.571164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.571137    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-data\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.571259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.571232    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-credential-socket\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.571539 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.571507    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istiod-ca-cert\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.571673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.571547    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-certs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.573153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.573134    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-podinfo\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.573362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.573342    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-envoy\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.595918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.595860    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-token\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.596184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.596160    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw6gs\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-kube-api-access-lw6gs\") pod \"e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") " pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:27.666143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:27.666115    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:28.003459 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:53:28.003425    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0928e6_229e_4d23_82b7_04042ffdc0c4.slice/crio-d3c6f9d2f89443f1484ae2ab5ac4bb0cc2d8c97918757c1805d287ce778b71d4 WatchSource:0}: Error finding container d3c6f9d2f89443f1484ae2ab5ac4bb0cc2d8c97918757c1805d287ce778b71d4: Status 404 returned error can't find the container with id d3c6f9d2f89443f1484ae2ab5ac4bb0cc2d8c97918757c1805d287ce778b71d4
Nov 02 14:53:28.004064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:28.004039    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"]
Nov 02 14:53:28.006545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:28.006505    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:53:28.006658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:28.006593    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:53:28.006658 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:28.006635    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:53:28.327966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:28.327870    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" event={"ID":"8f0928e6-229e-4d23-82b7-04042ffdc0c4","Type":"ContainerStarted","Data":"3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77"}
Nov 02 14:53:28.327966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:28.327931    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" event={"ID":"8f0928e6-229e-4d23-82b7-04042ffdc0c4","Type":"ContainerStarted","Data":"d3c6f9d2f89443f1484ae2ab5ac4bb0cc2d8c97918757c1805d287ce778b71d4"}
Nov 02 14:53:28.378742 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:28.378635    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" podStartSLOduration=1.37861171 podStartE2EDuration="1.37861171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:28.376047727 +0000 UTC m=+4199.127352190" watchObservedRunningTime="2026-11-02 14:53:28.37861171 +0000 UTC m=+4199.129916182"
Nov 02 14:53:28.666888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:28.666805    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:29.704508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:29.704120    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused" start-of-body=
Nov 02 14:53:29.704508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:29.704180    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused"
Nov 02 14:53:30.666903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:30.666868    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused" start-of-body=
Nov 02 14:53:30.667097 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:30.666923    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused"
Nov 02 14:53:30.996806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:30.993137    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:53:31.013813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:31.010033    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:53:31.667094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:31.667058    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused" start-of-body=
Nov 02 14:53:31.667271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:31.667110    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused"
Nov 02 14:53:32.291201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.291160    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"]
Nov 02 14:53:32.306498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.306469    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.325871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.325840    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:53:32.326153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.326135    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:53:32.326413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.326397    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:53:32.334799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.330605    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"]
Nov 02 14:53:32.396801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.375977    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:53:32.396801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.376865    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh885\" (UniqueName: \"kubernetes.io/projected/d32a0860-4a22-4b17-830e-7493e26abcb3-kube-api-access-qh885\") pod \"maas-api-revoke-sub-a4e0b26e-s4xnr\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.396801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.376914    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4e0b26e-s4xnr\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.396801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.377021    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4e0b26e-s4xnr\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.494885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.494847    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh885\" (UniqueName: \"kubernetes.io/projected/d32a0860-4a22-4b17-830e-7493e26abcb3-kube-api-access-qh885\") pod \"maas-api-revoke-sub-a4e0b26e-s4xnr\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.495097 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.494897    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4e0b26e-s4xnr\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.495097 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.494975    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4e0b26e-s4xnr\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.495806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.495754    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4e0b26e-s4xnr\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.497485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.497463    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4e0b26e-s4xnr\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.520231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.520203    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh885\" (UniqueName: \"kubernetes.io/projected/d32a0860-4a22-4b17-830e-7493e26abcb3-kube-api-access-qh885\") pod \"maas-api-revoke-sub-a4e0b26e-s4xnr\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.656857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.656750    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:32.667383 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.667356    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused" start-of-body=
Nov 02 14:53:32.667551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:32.667412    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused"
Nov 02 14:53:33.003450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:33.003418    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"]
Nov 02 14:53:33.004632 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:53:33.004607    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32a0860_4a22_4b17_830e_7493e26abcb3.slice/crio-a0a5ea8352adea245b41c60532845fb6a1dc02179129197eb6ecb4ae4eaa25f0 WatchSource:0}: Error finding container a0a5ea8352adea245b41c60532845fb6a1dc02179129197eb6ecb4ae4eaa25f0: Status 404 returned error can't find the container with id a0a5ea8352adea245b41c60532845fb6a1dc02179129197eb6ecb4ae4eaa25f0
Nov 02 14:53:33.410876 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:33.410841    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr" event={"ID":"d32a0860-4a22-4b17-830e-7493e26abcb3","Type":"ContainerStarted","Data":"a0a5ea8352adea245b41c60532845fb6a1dc02179129197eb6ecb4ae4eaa25f0"}
Nov 02 14:53:33.666551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:33.666471    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused" start-of-body=
Nov 02 14:53:33.666551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:33.666531    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused"
Nov 02 14:53:34.385228 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:53:34.385193    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32a0860_4a22_4b17_830e_7493e26abcb3.slice/crio-conmon-b55daf415eedc633c6db7ed9758ed65aad5a83cb3d366b04dbbb5ab317156f35.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 14:53:34.422391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:34.422355    5034 generic.go:356] "Generic (PLEG): container finished" podID="d32a0860-4a22-4b17-830e-7493e26abcb3" containerID="b55daf415eedc633c6db7ed9758ed65aad5a83cb3d366b04dbbb5ab317156f35" exitCode=0
Nov 02 14:53:34.422809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:34.422448    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr" event={"ID":"d32a0860-4a22-4b17-830e-7493e26abcb3","Type":"ContainerDied","Data":"b55daf415eedc633c6db7ed9758ed65aad5a83cb3d366b04dbbb5ab317156f35"}
Nov 02 14:53:34.667156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:34.667058    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused" start-of-body=
Nov 02 14:53:34.667156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:34.667118    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.157:15021/healthz/ready\": dial tcp 10.132.1.157:15021: connect: connection refused"
Nov 02 14:53:35.618621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.618593    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:35.672557 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.672526    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:35.672896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.672864    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:35.673666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.673632    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:53:35.745764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.745741    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32a0860-4a22-4b17-830e-7493e26abcb3-kube-api-access-qh885\" (UniqueName: \"kubernetes.io/projected/d32a0860-4a22-4b17-830e-7493e26abcb3-kube-api-access-qh885\") pod \"d32a0860-4a22-4b17-830e-7493e26abcb3\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") "
Nov 02 14:53:35.746034 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.746017    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-openshift-ca\") pod \"d32a0860-4a22-4b17-830e-7493e26abcb3\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") "
Nov 02 14:53:35.746124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.746059    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-xks-ca\") pod \"d32a0860-4a22-4b17-830e-7493e26abcb3\" (UID: \"d32a0860-4a22-4b17-830e-7493e26abcb3\") "
Nov 02 14:53:35.746398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.746367    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-openshift-ca" pod "d32a0860-4a22-4b17-830e-7493e26abcb3" (UID: "d32a0860-4a22-4b17-830e-7493e26abcb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:35.747951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.747930    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32a0860-4a22-4b17-830e-7493e26abcb3-kube-api-access-qh885" pod "d32a0860-4a22-4b17-830e-7493e26abcb3" (UID: "d32a0860-4a22-4b17-830e-7493e26abcb3"). InnerVolumeSpecName "kube-api-access-qh885". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:35.748130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.748110    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-xks-ca" pod "d32a0860-4a22-4b17-830e-7493e26abcb3" (UID: "d32a0860-4a22-4b17-830e-7493e26abcb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:35.848384 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.848345    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:35.848674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.848649    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32a0860-4a22-4b17-830e-7493e26abcb3-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:35.848740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:35.848683    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh885\" (UniqueName: \"kubernetes.io/projected/d32a0860-4a22-4b17-830e-7493e26abcb3-kube-api-access-qh885\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:36.440262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:36.440232    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"
Nov 02 14:53:36.440424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:36.440233    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr" event={"ID":"d32a0860-4a22-4b17-830e-7493e26abcb3","Type":"ContainerDied","Data":"a0a5ea8352adea245b41c60532845fb6a1dc02179129197eb6ecb4ae4eaa25f0"}
Nov 02 14:53:36.440424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:36.440369    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0a5ea8352adea245b41c60532845fb6a1dc02179129197eb6ecb4ae4eaa25f0"
Nov 02 14:53:39.405989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:39.405956    5034 scope.go:122] "RemoveContainer" containerID="58cef5ea54650dce0d3a7759745fe1a17f3bae4127b893970f5250b5a97f2338"
Nov 02 14:53:39.420269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:39.420246    5034 scope.go:122] "RemoveContainer" containerID="34e4067ce969c9cfcc59491fdde6fc69a59039eb9d39cc327ba314c171acd1ca"
Nov 02 14:53:39.439588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:39.439566    5034 scope.go:122] "RemoveContainer" containerID="66e1f4daef75ed576e9ffe6fab6a9f82521608a8a52980a91572dab15259d803"
Nov 02 14:53:41.399171 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.399139    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"]
Nov 02 14:53:41.399956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.399924    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d32a0860-4a22-4b17-830e-7493e26abcb3" containerName="revoke-keys"
Nov 02 14:53:41.399956 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.399946    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32a0860-4a22-4b17-830e-7493e26abcb3" containerName="revoke-keys"
Nov 02 14:53:41.400183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.400141    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32a0860-4a22-4b17-830e-7493e26abcb3" containerName="revoke-keys"
Nov 02 14:53:41.405620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.405598    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.424986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.424959    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"]
Nov 02 14:53:41.429015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.428984    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01232c80-sxkh6\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.429141 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.429119    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n69f\" (UniqueName: \"kubernetes.io/projected/a8a363cd-a08f-4ffe-8272-a6b7943407a7-kube-api-access-5n69f\") pod \"maas-api-revoke-sub-01232c80-sxkh6\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.429215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.429170    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01232c80-sxkh6\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.434029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.433989    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:53:41.434317 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.434295    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:53:41.434426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.434381    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:53:41.438716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.438697    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:53:41.530686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.530655    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01232c80-sxkh6\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.530875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.530718    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n69f\" (UniqueName: \"kubernetes.io/projected/a8a363cd-a08f-4ffe-8272-a6b7943407a7-kube-api-access-5n69f\") pod \"maas-api-revoke-sub-01232c80-sxkh6\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.530875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.530752    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01232c80-sxkh6\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.531450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.531424    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01232c80-sxkh6\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.533210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.533188    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01232c80-sxkh6\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.575064 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.575019    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n69f\" (UniqueName: \"kubernetes.io/projected/a8a363cd-a08f-4ffe-8272-a6b7943407a7-kube-api-access-5n69f\") pod \"maas-api-revoke-sub-01232c80-sxkh6\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:41.719578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:41.719514    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:42.028711 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:53:42.028682    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a363cd_a08f_4ffe_8272_a6b7943407a7.slice/crio-1f2d02b3756e1ed9ba45c7bfb37ae61d313b03863cb582d9e1cfe131eff9e4d7 WatchSource:0}: Error finding container 1f2d02b3756e1ed9ba45c7bfb37ae61d313b03863cb582d9e1cfe131eff9e4d7: Status 404 returned error can't find the container with id 1f2d02b3756e1ed9ba45c7bfb37ae61d313b03863cb582d9e1cfe131eff9e4d7
Nov 02 14:53:42.030940 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:42.030913    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"]
Nov 02 14:53:42.495564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:42.495528    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6" event={"ID":"a8a363cd-a08f-4ffe-8272-a6b7943407a7","Type":"ContainerStarted","Data":"96c1a2e1e65d8c5627895c86f93fb06b8a4917d686dfa09b206269c5ec3bceab"}
Nov 02 14:53:42.496057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:42.495596    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6" event={"ID":"a8a363cd-a08f-4ffe-8272-a6b7943407a7","Type":"ContainerStarted","Data":"1f2d02b3756e1ed9ba45c7bfb37ae61d313b03863cb582d9e1cfe131eff9e4d7"}
Nov 02 14:53:42.545718 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:42.545474    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6" podStartSLOduration=1.545457388 podStartE2EDuration="1.545457388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:42.545102734 +0000 UTC m=+4213.296407173" watchObservedRunningTime="2026-11-02 14:53:42.545457388 +0000 UTC m=+4213.296761827"
Nov 02 14:53:43.509047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:43.509004    5034 generic.go:356] "Generic (PLEG): container finished" podID="a8a363cd-a08f-4ffe-8272-a6b7943407a7" containerID="96c1a2e1e65d8c5627895c86f93fb06b8a4917d686dfa09b206269c5ec3bceab" exitCode=0
Nov 02 14:53:43.509537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:43.509102    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6" event={"ID":"a8a363cd-a08f-4ffe-8272-a6b7943407a7","Type":"ContainerDied","Data":"96c1a2e1e65d8c5627895c86f93fb06b8a4917d686dfa09b206269c5ec3bceab"}
Nov 02 14:53:44.694148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.694115    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:44.778825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.778716    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-openshift-ca\") pod \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") "
Nov 02 14:53:44.779125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.779101    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-xks-ca\") pod \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") "
Nov 02 14:53:44.779125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.779112    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-openshift-ca" pod "a8a363cd-a08f-4ffe-8272-a6b7943407a7" (UID: "a8a363cd-a08f-4ffe-8272-a6b7943407a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:53:44.779286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.779151    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a363cd-a08f-4ffe-8272-a6b7943407a7-kube-api-access-5n69f\" (UniqueName: \"kubernetes.io/projected/a8a363cd-a08f-4ffe-8272-a6b7943407a7-kube-api-access-5n69f\") pod \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\" (UID: \"a8a363cd-a08f-4ffe-8272-a6b7943407a7\") "
Nov 02 14:53:44.779717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.779692    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:44.781741 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.781716    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-xks-ca" pod "a8a363cd-a08f-4ffe-8272-a6b7943407a7" (UID: "a8a363cd-a08f-4ffe-8272-a6b7943407a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:53:44.781867 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.781822    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a363cd-a08f-4ffe-8272-a6b7943407a7-kube-api-access-5n69f" pod "a8a363cd-a08f-4ffe-8272-a6b7943407a7" (UID: "a8a363cd-a08f-4ffe-8272-a6b7943407a7"). InnerVolumeSpecName "kube-api-access-5n69f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:53:44.881329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.881295    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a363cd-a08f-4ffe-8272-a6b7943407a7-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:44.881329 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:44.881324    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n69f\" (UniqueName: \"kubernetes.io/projected/a8a363cd-a08f-4ffe-8272-a6b7943407a7-kube-api-access-5n69f\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:53:45.526533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:45.526504    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"
Nov 02 14:53:45.526676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:45.526541    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6" event={"ID":"a8a363cd-a08f-4ffe-8272-a6b7943407a7","Type":"ContainerDied","Data":"1f2d02b3756e1ed9ba45c7bfb37ae61d313b03863cb582d9e1cfe131eff9e4d7"}
Nov 02 14:53:45.526676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:45.526586    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f2d02b3756e1ed9ba45c7bfb37ae61d313b03863cb582d9e1cfe131eff9e4d7"
Nov 02 14:53:45.937894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:45.937862    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"]
Nov 02 14:53:48.415641 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:53:48.415600    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:53:56.602146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:56.602110    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"]
Nov 02 14:53:56.621254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:56.621215    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy" containerID="cri-o://55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484" gracePeriod=30
Nov 02 14:53:57.565307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.565270    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"]
Nov 02 14:53:57.566491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.566468    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a8a363cd-a08f-4ffe-8272-a6b7943407a7" containerName="revoke-keys"
Nov 02 14:53:57.566545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.566498    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a363cd-a08f-4ffe-8272-a6b7943407a7" containerName="revoke-keys"
Nov 02 14:53:57.566692 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.566678    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a363cd-a08f-4ffe-8272-a6b7943407a7" containerName="revoke-keys"
Nov 02 14:53:57.572307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.572281    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.588800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.588172    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"]
Nov 02 14:53:57.618130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.617047    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:53:57.618130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.617078    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:53:57.618130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.617191    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:53:57.630714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.630686    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:53:57.719547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.719511    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-894a26f8-7svbf\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.719887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.719859    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hn7\" (UniqueName: \"kubernetes.io/projected/8042d30f-134f-46bd-819c-218563ea6cc6-kube-api-access-h9hn7\") pod \"maas-api-revoke-sub-894a26f8-7svbf\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.720019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.720000    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-894a26f8-7svbf\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.821029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.820942    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-894a26f8-7svbf\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.821199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.821173    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hn7\" (UniqueName: \"kubernetes.io/projected/8042d30f-134f-46bd-819c-218563ea6cc6-kube-api-access-h9hn7\") pod \"maas-api-revoke-sub-894a26f8-7svbf\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.821273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.821249    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-894a26f8-7svbf\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.821650 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.821628    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-894a26f8-7svbf\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.823664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.823643    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-894a26f8-7svbf\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.846621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.846586    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hn7\" (UniqueName: \"kubernetes.io/projected/8042d30f-134f-46bd-819c-218563ea6cc6-kube-api-access-h9hn7\") pod \"maas-api-revoke-sub-894a26f8-7svbf\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:57.894161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:57.894127    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:53:58.135369 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.135283    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"]
Nov 02 14:53:58.140842 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.140803    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.163510 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.163485    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"]
Nov 02 14:53:58.224401 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:53:58.224369    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8042d30f_134f_46bd_819c_218563ea6cc6.slice/crio-69dd91b07168698c4df07487ad50d4fa3fbac3e0bdad99adac95cd1417b1b79a WatchSource:0}: Error finding container 69dd91b07168698c4df07487ad50d4fa3fbac3e0bdad99adac95cd1417b1b79a: Status 404 returned error can't find the container with id 69dd91b07168698c4df07487ad50d4fa3fbac3e0bdad99adac95cd1417b1b79a
Nov 02 14:53:58.227974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.227946    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"]
Nov 02 14:53:58.328544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.328511    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9ggz\" (UniqueName: \"kubernetes.io/projected/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-kube-api-access-p9ggz\") pod \"maas-api-revoke-sub-afaa709a-qrh6v\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.328694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.328652    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afaa709a-qrh6v\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.328822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.328804    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afaa709a-qrh6v\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.430502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.430470    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afaa709a-qrh6v\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.430675 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.430574    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afaa709a-qrh6v\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.430770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.430748    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9ggz\" (UniqueName: \"kubernetes.io/projected/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-kube-api-access-p9ggz\") pod \"maas-api-revoke-sub-afaa709a-qrh6v\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.431459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.431437    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afaa709a-qrh6v\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.433131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.433110    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afaa709a-qrh6v\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.455300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.455276    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9ggz\" (UniqueName: \"kubernetes.io/projected/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-kube-api-access-p9ggz\") pod \"maas-api-revoke-sub-afaa709a-qrh6v\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.474081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.474059    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:53:58.690677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.690573    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf" event={"ID":"8042d30f-134f-46bd-819c-218563ea6cc6","Type":"ContainerStarted","Data":"5086327a6ade5df3b49ca3fd504e2f92f7b2416a810af008adaff9cf64987235"}
Nov 02 14:53:58.691187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.690683    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf" event={"ID":"8042d30f-134f-46bd-819c-218563ea6cc6","Type":"ContainerStarted","Data":"69dd91b07168698c4df07487ad50d4fa3fbac3e0bdad99adac95cd1417b1b79a"}
Nov 02 14:53:58.740790 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.740685    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf" podStartSLOduration=1.740666475 podStartE2EDuration="1.740666475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:58.737444182 +0000 UTC m=+4229.488748620" watchObservedRunningTime="2026-11-02 14:53:58.740666475 +0000 UTC m=+4229.491970914"
Nov 02 14:53:58.826900 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:53:58.826818    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b6c22d_86cd_4f3f_bc9a_0bbdd0da8586.slice/crio-bff2d80367a26bf7197e95e9e45b49f702ebf6f1c949f109d44378b5c2b8ac83 WatchSource:0}: Error finding container bff2d80367a26bf7197e95e9e45b49f702ebf6f1c949f109d44378b5c2b8ac83: Status 404 returned error can't find the container with id bff2d80367a26bf7197e95e9e45b49f702ebf6f1c949f109d44378b5c2b8ac83
Nov 02 14:53:58.827852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:58.827821    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"]
Nov 02 14:53:59.700201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:59.700106    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v" event={"ID":"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586","Type":"ContainerStarted","Data":"08852bc60cb728523fe56e60fd8717331d3751f369656b3bfa13e14f364aa9a4"}
Nov 02 14:53:59.700201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:59.700174    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v" event={"ID":"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586","Type":"ContainerStarted","Data":"bff2d80367a26bf7197e95e9e45b49f702ebf6f1c949f109d44378b5c2b8ac83"}
Nov 02 14:53:59.702812 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:59.702747    5034 generic.go:356] "Generic (PLEG): container finished" podID="8042d30f-134f-46bd-819c-218563ea6cc6" containerID="5086327a6ade5df3b49ca3fd504e2f92f7b2416a810af008adaff9cf64987235" exitCode=0
Nov 02 14:53:59.703075 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:59.703052    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf" event={"ID":"8042d30f-134f-46bd-819c-218563ea6cc6","Type":"ContainerDied","Data":"5086327a6ade5df3b49ca3fd504e2f92f7b2416a810af008adaff9cf64987235"}
Nov 02 14:53:59.743438 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:53:59.743378    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v" podStartSLOduration=1.743361214 podStartE2EDuration="1.743361214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:53:59.742836752 +0000 UTC m=+4230.494141191" watchObservedRunningTime="2026-11-02 14:53:59.743361214 +0000 UTC m=+4230.494665658"
Nov 02 14:54:00.118811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.118762    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:54:00.119022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.118838    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:54:00.713007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.712960    5034 generic.go:356] "Generic (PLEG): container finished" podID="c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586" containerID="08852bc60cb728523fe56e60fd8717331d3751f369656b3bfa13e14f364aa9a4" exitCode=0
Nov 02 14:54:00.713519 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.713095    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v" event={"ID":"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586","Type":"ContainerDied","Data":"08852bc60cb728523fe56e60fd8717331d3751f369656b3bfa13e14f364aa9a4"}
Nov 02 14:54:00.907107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.907076    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:54:00.975772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.975687    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main" containerID="cri-o://13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17" gracePeriod=45
Nov 02 14:54:00.976198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.976087    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-xks-ca\") pod \"8042d30f-134f-46bd-819c-218563ea6cc6\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") "
Nov 02 14:54:00.976277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.976209    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8042d30f-134f-46bd-819c-218563ea6cc6-kube-api-access-h9hn7\" (UniqueName: \"kubernetes.io/projected/8042d30f-134f-46bd-819c-218563ea6cc6-kube-api-access-h9hn7\") pod \"8042d30f-134f-46bd-819c-218563ea6cc6\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") "
Nov 02 14:54:00.976277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.976257    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-openshift-ca\") pod \"8042d30f-134f-46bd-819c-218563ea6cc6\" (UID: \"8042d30f-134f-46bd-819c-218563ea6cc6\") "
Nov 02 14:54:00.977222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.977168    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-openshift-ca" pod "8042d30f-134f-46bd-819c-218563ea6cc6" (UID: "8042d30f-134f-46bd-819c-218563ea6cc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:00.979262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.979231    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-xks-ca" pod "8042d30f-134f-46bd-819c-218563ea6cc6" (UID: "8042d30f-134f-46bd-819c-218563ea6cc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:00.979656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:00.979631    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8042d30f-134f-46bd-819c-218563ea6cc6-kube-api-access-h9hn7" pod "8042d30f-134f-46bd-819c-218563ea6cc6" (UID: "8042d30f-134f-46bd-819c-218563ea6cc6"). InnerVolumeSpecName "kube-api-access-h9hn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:01.078418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.078389    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.078535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.078423    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hn7\" (UniqueName: \"kubernetes.io/projected/8042d30f-134f-46bd-819c-218563ea6cc6-kube-api-access-h9hn7\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.078535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.078443    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8042d30f-134f-46bd-819c-218563ea6cc6-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.279395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.279364    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:54:01.385327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.385291    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6949af01-2c4c-4d4d-9bd5-02724101b868-kube-api-access-mqx7d\" (UniqueName: \"kubernetes.io/projected/6949af01-2c4c-4d4d-9bd5-02724101b868-kube-api-access-mqx7d\") pod \"6949af01-2c4c-4d4d-9bd5-02724101b868\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") "
Nov 02 14:54:01.385543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.385417    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6949af01-2c4c-4d4d-9bd5-02724101b868-tls-certs\" (UniqueName: \"kubernetes.io/secret/6949af01-2c4c-4d4d-9bd5-02724101b868-tls-certs\") pod \"6949af01-2c4c-4d4d-9bd5-02724101b868\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") "
Nov 02 14:54:01.385543 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.385516    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-home\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-home\") pod \"6949af01-2c4c-4d4d-9bd5-02724101b868\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") "
Nov 02 14:54:01.385679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.385645    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-model-cache\") pod \"6949af01-2c4c-4d4d-9bd5-02724101b868\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") "
Nov 02 14:54:01.385679 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.385676    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-dshm\") pod \"6949af01-2c4c-4d4d-9bd5-02724101b868\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") "
Nov 02 14:54:01.385885 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.385763    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-tmp-dir\") pod \"6949af01-2c4c-4d4d-9bd5-02724101b868\" (UID: \"6949af01-2c4c-4d4d-9bd5-02724101b868\") "
Nov 02 14:54:01.386388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.386351    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-home" pod "6949af01-2c4c-4d4d-9bd5-02724101b868" (UID: "6949af01-2c4c-4d4d-9bd5-02724101b868"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.386579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.386555    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-tmp-dir" pod "6949af01-2c4c-4d4d-9bd5-02724101b868" (UID: "6949af01-2c4c-4d4d-9bd5-02724101b868"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.391813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.387914    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-model-cache" pod "6949af01-2c4c-4d4d-9bd5-02724101b868" (UID: "6949af01-2c4c-4d4d-9bd5-02724101b868"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.391813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.390276    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6949af01-2c4c-4d4d-9bd5-02724101b868-kube-api-access-mqx7d" pod "6949af01-2c4c-4d4d-9bd5-02724101b868" (UID: "6949af01-2c4c-4d4d-9bd5-02724101b868"). InnerVolumeSpecName "kube-api-access-mqx7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:01.395947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.395909    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6949af01-2c4c-4d4d-9bd5-02724101b868-tls-certs" pod "6949af01-2c4c-4d4d-9bd5-02724101b868" (UID: "6949af01-2c4c-4d4d-9bd5-02724101b868"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:01.395947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.395923    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-dshm" pod "6949af01-2c4c-4d4d-9bd5-02724101b868" (UID: "6949af01-2c4c-4d4d-9bd5-02724101b868"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:01.488265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.488230    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.488265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.488260    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.488265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.488269    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.488578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.488281    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6949af01-2c4c-4d4d-9bd5-02724101b868-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.488578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.488295    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqx7d\" (UniqueName: \"kubernetes.io/projected/6949af01-2c4c-4d4d-9bd5-02724101b868-kube-api-access-mqx7d\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.488578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.488306    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6949af01-2c4c-4d4d-9bd5-02724101b868-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:01.723238 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.723211    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"
Nov 02 14:54:01.723592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.723248    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf" event={"ID":"8042d30f-134f-46bd-819c-218563ea6cc6","Type":"ContainerDied","Data":"69dd91b07168698c4df07487ad50d4fa3fbac3e0bdad99adac95cd1417b1b79a"}
Nov 02 14:54:01.723592 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.723296    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69dd91b07168698c4df07487ad50d4fa3fbac3e0bdad99adac95cd1417b1b79a"
Nov 02 14:54:01.725011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.724985    5034 generic.go:356] "Generic (PLEG): container finished" podID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerID="13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17" exitCode=0
Nov 02 14:54:01.725147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.725033    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"
Nov 02 14:54:01.725147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.725079    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" event={"ID":"6949af01-2c4c-4d4d-9bd5-02724101b868","Type":"ContainerDied","Data":"13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17"}
Nov 02 14:54:01.725147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.725122    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm" event={"ID":"6949af01-2c4c-4d4d-9bd5-02724101b868","Type":"ContainerDied","Data":"0412fa698b03801a883f52cd52894762eec3c08d4177a2216f56e6973f3358fe"}
Nov 02 14:54:01.725147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.725146    5034 scope.go:122] "RemoveContainer" containerID="13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17"
Nov 02 14:54:01.737254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.737232    5034 scope.go:122] "RemoveContainer" containerID="13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17"
Nov 02 14:54:01.737591 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:54:01.737567    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17\": container with ID starting with 13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17 not found: ID does not exist" containerID="13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17"
Nov 02 14:54:01.737663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.737604    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17"} err="failed to get container status \"13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17\": rpc error: code = NotFound desc = could not find container \"13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17\": container with ID starting with 13a9b8694eecafb15317e9071d95551618ce3495424931c75de7b7051b748c17 not found: ID does not exist"
Nov 02 14:54:01.810767 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.810708    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"]
Nov 02 14:54:01.817228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.817195    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1a562c/premium-w0-1a562c-kserve-5c7c64476-5pdrm"]
Nov 02 14:54:01.941367 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.941188    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:54:01.981269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.981243    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:54:01.996808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.995147    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-openshift-ca\") pod \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") "
Nov 02 14:54:01.996808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.995239    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-xks-ca\") pod \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") "
Nov 02 14:54:01.996808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.995310    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-kube-api-access-p9ggz\" (UniqueName: \"kubernetes.io/projected/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-kube-api-access-p9ggz\") pod \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\" (UID: \"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586\") "
Nov 02 14:54:02.001331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:01.998702    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-openshift-ca" pod "c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586" (UID: "c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:02.001461 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.001391    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-kube-api-access-p9ggz" pod "c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586" (UID: "c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586"). InnerVolumeSpecName "kube-api-access-p9ggz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:02.002633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.002604    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-xks-ca" pod "c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586" (UID: "c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:02.116289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.115248    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-certs\") pod \"af80e624-1231-4cfd-8024-64373c76b18d\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") "
Nov 02 14:54:02.116289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.115397    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af80e624-1231-4cfd-8024-64373c76b18d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af80e624-1231-4cfd-8024-64373c76b18d-istiod-ca-cert\") pod \"af80e624-1231-4cfd-8024-64373c76b18d\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") "
Nov 02 14:54:02.116289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.115435    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-envoy\") pod \"af80e624-1231-4cfd-8024-64373c76b18d\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") "
Nov 02 14:54:02.116289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.115524    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af80e624-1231-4cfd-8024-64373c76b18d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af80e624-1231-4cfd-8024-64373c76b18d-istio-podinfo\") pod \"af80e624-1231-4cfd-8024-64373c76b18d\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") "
Nov 02 14:54:02.116289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.115557    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-istio-token\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-istio-token\") pod \"af80e624-1231-4cfd-8024-64373c76b18d\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") "
Nov 02 14:54:02.116289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.115645    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-kube-api-access-npxw5\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-kube-api-access-npxw5\") pod \"af80e624-1231-4cfd-8024-64373c76b18d\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") "
Nov 02 14:54:02.116289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.115684    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-data\") pod \"af80e624-1231-4cfd-8024-64373c76b18d\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") "
Nov 02 14:54:02.116289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.115770    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-socket\") pod \"af80e624-1231-4cfd-8024-64373c76b18d\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") "
Nov 02 14:54:02.116289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.115844    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-credential-socket\") pod \"af80e624-1231-4cfd-8024-64373c76b18d\" (UID: \"af80e624-1231-4cfd-8024-64373c76b18d\") "
Nov 02 14:54:02.117435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.117404    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.117565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.117439    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9ggz\" (UniqueName: \"kubernetes.io/projected/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-kube-api-access-p9ggz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.117565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.117454    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.117754 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.117732    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-credential-socket" pod "af80e624-1231-4cfd-8024-64373c76b18d" (UID: "af80e624-1231-4cfd-8024-64373c76b18d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:02.118224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.118049    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-certs" pod "af80e624-1231-4cfd-8024-64373c76b18d" (UID: "af80e624-1231-4cfd-8024-64373c76b18d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:02.118756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.118469    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af80e624-1231-4cfd-8024-64373c76b18d-istiod-ca-cert" pod "af80e624-1231-4cfd-8024-64373c76b18d" (UID: "af80e624-1231-4cfd-8024-64373c76b18d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:02.122131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.122107    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-socket" pod "af80e624-1231-4cfd-8024-64373c76b18d" (UID: "af80e624-1231-4cfd-8024-64373c76b18d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:02.124617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.124551    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-data" pod "af80e624-1231-4cfd-8024-64373c76b18d" (UID: "af80e624-1231-4cfd-8024-64373c76b18d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:02.126002 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.125979    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"]
Nov 02 14:54:02.127853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.127460    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af80e624-1231-4cfd-8024-64373c76b18d-istio-podinfo" pod "af80e624-1231-4cfd-8024-64373c76b18d" (UID: "af80e624-1231-4cfd-8024-64373c76b18d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:54:02.129146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.129119    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-istio-token" pod "af80e624-1231-4cfd-8024-64373c76b18d" (UID: "af80e624-1231-4cfd-8024-64373c76b18d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:02.129270 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.129248    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-kube-api-access-npxw5" pod "af80e624-1231-4cfd-8024-64373c76b18d" (UID: "af80e624-1231-4cfd-8024-64373c76b18d"). InnerVolumeSpecName "kube-api-access-npxw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:02.130225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.130202    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-envoy" pod "af80e624-1231-4cfd-8024-64373c76b18d" (UID: "af80e624-1231-4cfd-8024-64373c76b18d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:02.219106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.219029    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af80e624-1231-4cfd-8024-64373c76b18d-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.219106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.219062    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.219106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.219071    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npxw5\" (UniqueName: \"kubernetes.io/projected/af80e624-1231-4cfd-8024-64373c76b18d-kube-api-access-npxw5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.219106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.219080    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.219106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.219088    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.219106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.219096    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.219106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.219104    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.219106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.219112    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af80e624-1231-4cfd-8024-64373c76b18d-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.219463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.219119    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af80e624-1231-4cfd-8024-64373c76b18d-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:02.734222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.734184    5034 generic.go:356] "Generic (PLEG): container finished" podID="af80e624-1231-4cfd-8024-64373c76b18d" containerID="55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484" exitCode=0
Nov 02 14:54:02.734701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.734235    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"
Nov 02 14:54:02.734701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.734269    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" event={"ID":"af80e624-1231-4cfd-8024-64373c76b18d","Type":"ContainerDied","Data":"55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484"}
Nov 02 14:54:02.734701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.734321    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb" event={"ID":"af80e624-1231-4cfd-8024-64373c76b18d","Type":"ContainerDied","Data":"244a563a26492f269a52f20e82cea6953ecd67abd5734d4f6181a279fa63d558"}
Nov 02 14:54:02.734701 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.734345    5034 scope.go:122] "RemoveContainer" containerID="55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484"
Nov 02 14:54:02.737108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.737087    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"
Nov 02 14:54:02.737108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.737097    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v" event={"ID":"c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586","Type":"ContainerDied","Data":"bff2d80367a26bf7197e95e9e45b49f702ebf6f1c949f109d44378b5c2b8ac83"}
Nov 02 14:54:02.737320 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.737144    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff2d80367a26bf7197e95e9e45b49f702ebf6f1c949f109d44378b5c2b8ac83"
Nov 02 14:54:02.749250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.749211    5034 scope.go:122] "RemoveContainer" containerID="db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca"
Nov 02 14:54:02.763342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.763322    5034 scope.go:122] "RemoveContainer" containerID="55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484"
Nov 02 14:54:02.763651 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:54:02.763629    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484\": container with ID starting with 55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484 not found: ID does not exist" containerID="55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484"
Nov 02 14:54:02.763722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.763659    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484"} err="failed to get container status \"55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484\": rpc error: code = NotFound desc = could not find container \"55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484\": container with ID starting with 55bb69e5c5f46ec7b5adb24b4566f5139fe25be338d32cdb92d00776a9a3e484 not found: ID does not exist"
Nov 02 14:54:02.763722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.763677    5034 scope.go:122] "RemoveContainer" containerID="db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca"
Nov 02 14:54:02.764019 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:54:02.763993    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca\": container with ID starting with db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca not found: ID does not exist" containerID="db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca"
Nov 02 14:54:02.764070 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.764027    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca"} err="failed to get container status \"db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca\": rpc error: code = NotFound desc = could not find container \"db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca\": container with ID starting with db2c22027f355edc4ec98d8b0b0d253e248e2c0ca1ff5fc35f3b3fd71cc5b8ca not found: ID does not exist"
Nov 02 14:54:02.803396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.803363    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"]
Nov 02 14:54:02.809462 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:02.809434    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-662e3e-openshift-default-5b485f76d9-tk7bb"]
Nov 02 14:54:03.395809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:03.395762    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" path="/var/lib/kubelet/pods/6949af01-2c4c-4d4d-9bd5-02724101b868/volumes"
Nov 02 14:54:03.396326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:03.396309    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af80e624-1231-4cfd-8024-64373c76b18d" path="/var/lib/kubelet/pods/af80e624-1231-4cfd-8024-64373c76b18d/volumes"
Nov 02 14:54:08.867960 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.867920    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"]
Nov 02 14:54:08.879553 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.879521    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main"
Nov 02 14:54:08.879740 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.879726    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main"
Nov 02 14:54:08.879912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.879900    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586" containerName="revoke-keys"
Nov 02 14:54:08.879999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.879986    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586" containerName="revoke-keys"
Nov 02 14:54:08.880094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.880081    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8042d30f-134f-46bd-819c-218563ea6cc6" containerName="revoke-keys"
Nov 02 14:54:08.880299 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.880235    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8042d30f-134f-46bd-819c-218563ea6cc6" containerName="revoke-keys"
Nov 02 14:54:08.880574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.880542    5034 cpu_manager.go:414] "containerMap: removing container" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy"
Nov 02 14:54:08.880574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.880564    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy"
Nov 02 14:54:08.881004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.880600    5034 cpu_manager.go:414] "containerMap: removing container" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy"
Nov 02 14:54:08.881004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.880607    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy"
Nov 02 14:54:08.881130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.881083    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy"
Nov 02 14:54:08.881130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.881108    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6949af01-2c4c-4d4d-9bd5-02724101b868" containerName="main"
Nov 02 14:54:08.881229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.881127    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586" containerName="revoke-keys"
Nov 02 14:54:08.881229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.881186    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8042d30f-134f-46bd-819c-218563ea6cc6" containerName="revoke-keys"
Nov 02 14:54:08.885322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.885301    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:08.893265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.893239    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"]
Nov 02 14:54:08.903400 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.902267    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:54:08.915621 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.915599    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:54:08.915749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.915725    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:54:08.915749 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.915730    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:54:08.940839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:08.940813    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"]
Nov 02 14:54:09.020763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.020727    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8p5\" (UniqueName: \"kubernetes.io/projected/cf9e0676-6e51-4517-90ee-4c945a632e57-kube-api-access-jw8p5\") pod \"maas-api-revoke-sub-e07dbdd7-2vv77\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.020965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.020793    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07dbdd7-2vv77\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.021057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.021019    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07dbdd7-2vv77\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.122720 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.122642    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07dbdd7-2vv77\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.122909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.122764    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8p5\" (UniqueName: \"kubernetes.io/projected/cf9e0676-6e51-4517-90ee-4c945a632e57-kube-api-access-jw8p5\") pod \"maas-api-revoke-sub-e07dbdd7-2vv77\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.122909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.122827    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07dbdd7-2vv77\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.123344 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.123321    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e07dbdd7-2vv77\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.125201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.125172    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e07dbdd7-2vv77\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.147897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.147871    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8p5\" (UniqueName: \"kubernetes.io/projected/cf9e0676-6e51-4517-90ee-4c945a632e57-kube-api-access-jw8p5\") pod \"maas-api-revoke-sub-e07dbdd7-2vv77\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.222579 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.222548    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:09.605237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.595088    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"]
Nov 02 14:54:09.861275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:09.861200    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77" event={"ID":"cf9e0676-6e51-4517-90ee-4c945a632e57","Type":"ContainerStarted","Data":"07b73af09710ebaffa57a43d7c3f0ebc06dd4bc4cf49c43ff9b2137ea02f3422"}
Nov 02 14:54:10.870142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:10.870108    5034 generic.go:356] "Generic (PLEG): container finished" podID="cf9e0676-6e51-4517-90ee-4c945a632e57" containerID="f6048b0c07431c459b611b47cee369ab1ea1fac1a125fb7e006ec0a16c32998d" exitCode=0
Nov 02 14:54:10.870582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:10.870198    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77" event={"ID":"cf9e0676-6e51-4517-90ee-4c945a632e57","Type":"ContainerDied","Data":"f6048b0c07431c459b611b47cee369ab1ea1fac1a125fb7e006ec0a16c32998d"}
Nov 02 14:54:12.065301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.065275    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:12.178239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.178150    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-openshift-ca\") pod \"cf9e0676-6e51-4517-90ee-4c945a632e57\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") "
Nov 02 14:54:12.178239 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.178211    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-xks-ca\") pod \"cf9e0676-6e51-4517-90ee-4c945a632e57\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") "
Nov 02 14:54:12.178477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.178251    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9e0676-6e51-4517-90ee-4c945a632e57-kube-api-access-jw8p5\" (UniqueName: \"kubernetes.io/projected/cf9e0676-6e51-4517-90ee-4c945a632e57-kube-api-access-jw8p5\") pod \"cf9e0676-6e51-4517-90ee-4c945a632e57\" (UID: \"cf9e0676-6e51-4517-90ee-4c945a632e57\") "
Nov 02 14:54:12.178541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.178503    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-openshift-ca" pod "cf9e0676-6e51-4517-90ee-4c945a632e57" (UID: "cf9e0676-6e51-4517-90ee-4c945a632e57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:12.178875 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.178855    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:12.180466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.180430    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-xks-ca" pod "cf9e0676-6e51-4517-90ee-4c945a632e57" (UID: "cf9e0676-6e51-4517-90ee-4c945a632e57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:12.180549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.180492    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9e0676-6e51-4517-90ee-4c945a632e57-kube-api-access-jw8p5" pod "cf9e0676-6e51-4517-90ee-4c945a632e57" (UID: "cf9e0676-6e51-4517-90ee-4c945a632e57"). InnerVolumeSpecName "kube-api-access-jw8p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:12.280118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.280083    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf9e0676-6e51-4517-90ee-4c945a632e57-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:12.280118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.280115    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw8p5\" (UniqueName: \"kubernetes.io/projected/cf9e0676-6e51-4517-90ee-4c945a632e57-kube-api-access-jw8p5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:12.887938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.887906    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"
Nov 02 14:54:12.888145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.887947    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77" event={"ID":"cf9e0676-6e51-4517-90ee-4c945a632e57","Type":"ContainerDied","Data":"07b73af09710ebaffa57a43d7c3f0ebc06dd4bc4cf49c43ff9b2137ea02f3422"}
Nov 02 14:54:12.888145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:12.887996    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07b73af09710ebaffa57a43d7c3f0ebc06dd4bc4cf49c43ff9b2137ea02f3422"
Nov 02 14:54:13.941454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:13.941400    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z" podUID="c619c67a-04e0-45a2-8759-4f278b5b3326" containerName="payload-processing" containerID="cri-o://109e2d75d713964eb4bb00d30d4ac07cf9e4608c9a27a3798e2ecfacdc5dcf05" gracePeriod=25
Nov 02 14:54:17.150627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.150589    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" containerID="cri-o://6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b" gracePeriod=45
Nov 02 14:54:17.442276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.442245    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:54:17.551828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.551221    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tls-certs\" (UniqueName: \"kubernetes.io/secret/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tls-certs\") pod \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") "
Nov 02 14:54:17.551828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.551348    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tmp-dir\") pod \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") "
Nov 02 14:54:17.551828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.551475    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-model-cache\") pod \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") "
Nov 02 14:54:17.551828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.551527    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-kube-api-access-z7ltd\" (UniqueName: \"kubernetes.io/projected/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-kube-api-access-z7ltd\") pod \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") "
Nov 02 14:54:17.551828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.551631    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-dshm\") pod \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") "
Nov 02 14:54:17.551828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.551625    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tmp-dir" pod "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" (UID: "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:17.551828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.551711    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-home\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-home\") pod \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\" (UID: \"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68\") "
Nov 02 14:54:17.551828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.551714    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-model-cache" pod "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" (UID: "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:17.552418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.551991    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-home" pod "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" (UID: "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:17.553440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.553421    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:17.553586 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.553573    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:17.553680 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.553667    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:17.554157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.554127    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-dshm" pod "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" (UID: "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:17.554254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.554154    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tls-certs" pod "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" (UID: "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:54:17.554254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.554237    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-kube-api-access-z7ltd" pod "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" (UID: "20351fa4-e1a5-4e28-b2fe-750b5fbf8a68"). InnerVolumeSpecName "kube-api-access-z7ltd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:17.655091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.655052    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7ltd\" (UniqueName: \"kubernetes.io/projected/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-kube-api-access-z7ltd\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:17.655091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.655091    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:17.655364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.655107    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:17.871311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.871277    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"]
Nov 02 14:54:17.872073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.872052    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cf9e0676-6e51-4517-90ee-4c945a632e57" containerName="revoke-keys"
Nov 02 14:54:17.872073 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.872071    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9e0676-6e51-4517-90ee-4c945a632e57" containerName="revoke-keys"
Nov 02 14:54:17.872248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.872083    5034 cpu_manager.go:414] "containerMap: removing container" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main"
Nov 02 14:54:17.872248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.872089    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main"
Nov 02 14:54:17.872248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.872204    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9e0676-6e51-4517-90ee-4c945a632e57" containerName="revoke-keys"
Nov 02 14:54:17.872248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.872217    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main"
Nov 02 14:54:17.872248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.872228    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="af80e624-1231-4cfd-8024-64373c76b18d" containerName="istio-proxy"
Nov 02 14:54:17.876416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.876387    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:17.907524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.907491    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-2cddf2-openshift-default-dockercfg-8zz64\""
Nov 02 14:54:17.914356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.914325    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"]
Nov 02 14:54:17.931974 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.931942    5034 generic.go:356] "Generic (PLEG): container finished" podID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerID="6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b" exitCode=0
Nov 02 14:54:17.932123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.931986    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"
Nov 02 14:54:17.932123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.932025    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" event={"ID":"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68","Type":"ContainerDied","Data":"6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b"}
Nov 02 14:54:17.932123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.932065    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" event={"ID":"20351fa4-e1a5-4e28-b2fe-750b5fbf8a68","Type":"ContainerDied","Data":"73b02223f73acdf6f086824ff2cc677dd02593a42f4c45740135bd5e9aadcaf5"}
Nov 02 14:54:17.932123 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.932086    5034 scope.go:122] "RemoveContainer" containerID="6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b"
Nov 02 14:54:17.946193 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.946178    5034 scope.go:122] "RemoveContainer" containerID="6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b"
Nov 02 14:54:17.946502 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:54:17.946480    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b\": container with ID starting with 6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b not found: ID does not exist" containerID="6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b"
Nov 02 14:54:17.946594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.946512    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b"} err="failed to get container status \"6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b\": rpc error: code = NotFound desc = could not find container \"6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b\": container with ID starting with 6821dc6cd21a06a040a6a1d6887b5e5cea8364d37321b1d4e792808c138bae6b not found: ID does not exist"
Nov 02 14:54:17.981558 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.981531    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z" podUID="c619c67a-04e0-45a2-8759-4f278b5b3326" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:54:17.993941 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:17.993910    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"]
Nov 02 14:54:18.000240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.000213    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg"]
Nov 02 14:54:18.060315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.060289    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-data\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.060435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.060355    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-credential-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.060491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.060472    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-podinfo\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.060533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.060503    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-token\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.060570 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.060529    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qvdr\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-kube-api-access-8qvdr\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.060671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.060650    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-envoy\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.060711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.060698    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-certs\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.060849 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.060838    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.060891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.060865    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istiod-ca-cert\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162172    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-envoy\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162219    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-certs\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162300    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162326    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istiod-ca-cert\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162401    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-data\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162449    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-credential-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162495    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-podinfo\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162528    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-token\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162566    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qvdr\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-kube-api-access-8qvdr\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162682    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-certs\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.162752 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162730    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.163274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.162829    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-data\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.163274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.163049    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-credential-socket\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.163274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.163252    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istiod-ca-cert\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.164862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.164842    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-envoy\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.165247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.165226    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-podinfo\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.187877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.187849    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-token\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.187999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.187881    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qvdr\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-kube-api-access-8qvdr\") pod \"e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") " pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.190764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.190741    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:18.343802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.343750    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f30173/distinct-2-w3-f30173-kserve-7679bb87-cpjcg" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" containerName="main" probeResult="failure" output="Get \"https://10.132.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:54:18.415892 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:54:18.415810    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:54:18.517545 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:54:18.517516    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81deadf7_ae3c_4c10_a7e1_40906f764a9f.slice/crio-33cac51c769d40114be3f76ec2b9b37a1f6cbad3341775159906f75a46af2d38 WatchSource:0}: Error finding container 33cac51c769d40114be3f76ec2b9b37a1f6cbad3341775159906f75a46af2d38: Status 404 returned error can't find the container with id 33cac51c769d40114be3f76ec2b9b37a1f6cbad3341775159906f75a46af2d38
Nov 02 14:54:18.518883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.518861    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"]
Nov 02 14:54:18.520155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.520122    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:54:18.520235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.520185    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:54:18.520235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.520215    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:54:18.942211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.942170    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" event={"ID":"81deadf7-ae3c-4c10-a7e1-40906f764a9f","Type":"ContainerStarted","Data":"9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9"}
Nov 02 14:54:18.942416 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.942239    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" event={"ID":"81deadf7-ae3c-4c10-a7e1-40906f764a9f","Type":"ContainerStarted","Data":"33cac51c769d40114be3f76ec2b9b37a1f6cbad3341775159906f75a46af2d38"}
Nov 02 14:54:18.996887 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:18.996826    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" podStartSLOduration=1.9968081199999999 podStartE2EDuration="1.99680812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:54:18.990934021 +0000 UTC m=+4249.742238458" watchObservedRunningTime="2026-11-02 14:54:18.99680812 +0000 UTC m=+4249.748112559"
Nov 02 14:54:19.200859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:19.191343    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:19.200859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:19.196350    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Nov 02 14:54:19.200859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:19.196410    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Nov 02 14:54:19.436802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:19.436446    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20351fa4-e1a5-4e28-b2fe-750b5fbf8a68" path="/var/lib/kubelet/pods/20351fa4-e1a5-4e28-b2fe-750b5fbf8a68/volumes"
Nov 02 14:54:20.196803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:20.195768    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Nov 02 14:54:20.196803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:20.195843    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Nov 02 14:54:21.192800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:21.192105    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Nov 02 14:54:21.192800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:21.192169    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Nov 02 14:54:22.194860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:22.194140    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Nov 02 14:54:22.194860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:22.194201    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Nov 02 14:54:23.200213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:23.200170    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Nov 02 14:54:23.200619 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:23.200241    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Nov 02 14:54:24.193803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:24.193151    5034 patch_prober.go:29] interesting pod/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused" start-of-body=
Nov 02 14:54:24.193803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:24.193214    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.163:15021/healthz/ready\": dial tcp 10.132.1.163:15021: connect: connection refused"
Nov 02 14:54:25.212843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:25.210761    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:25.212843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:25.212443    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:25.213925 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:25.213888    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:54:26.037267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:26.037232    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"]
Nov 02 14:54:26.050822 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:26.050789    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf0c8306-mxbpw"]
Nov 02 14:54:27.401226 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:27.401190    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d47e5f-b43d-43b9-bab0-4a71b7023f14" path="/var/lib/kubelet/pods/65d47e5f-b43d-43b9-bab0-4a71b7023f14/volumes"
Nov 02 14:54:27.981624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:27.981577    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z" podUID="c619c67a-04e0-45a2-8759-4f278b5b3326" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:54:34.188324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:34.188128    5034 generic.go:356] "Generic (PLEG): container finished" podID="c619c67a-04e0-45a2-8759-4f278b5b3326" containerID="109e2d75d713964eb4bb00d30d4ac07cf9e4608c9a27a3798e2ecfacdc5dcf05" exitCode=0
Nov 02 14:54:34.188324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:34.188237    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z" event={"ID":"c619c67a-04e0-45a2-8759-4f278b5b3326","Type":"ContainerDied","Data":"109e2d75d713964eb4bb00d30d4ac07cf9e4608c9a27a3798e2ecfacdc5dcf05"}
Nov 02 14:54:34.258316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:34.258290    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 14:54:34.299229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:34.299191    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c619c67a-04e0-45a2-8759-4f278b5b3326-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c619c67a-04e0-45a2-8759-4f278b5b3326-plugins-config-volume\") pod \"c619c67a-04e0-45a2-8759-4f278b5b3326\" (UID: \"c619c67a-04e0-45a2-8759-4f278b5b3326\") "
Nov 02 14:54:34.299406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:34.299297    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c619c67a-04e0-45a2-8759-4f278b5b3326-kube-api-access-mnv4h\" (UniqueName: \"kubernetes.io/projected/c619c67a-04e0-45a2-8759-4f278b5b3326-kube-api-access-mnv4h\") pod \"c619c67a-04e0-45a2-8759-4f278b5b3326\" (UID: \"c619c67a-04e0-45a2-8759-4f278b5b3326\") "
Nov 02 14:54:34.300182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:34.300146    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c619c67a-04e0-45a2-8759-4f278b5b3326-plugins-config-volume" pod "c619c67a-04e0-45a2-8759-4f278b5b3326" (UID: "c619c67a-04e0-45a2-8759-4f278b5b3326"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:34.302856 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:34.302828    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c619c67a-04e0-45a2-8759-4f278b5b3326-kube-api-access-mnv4h" pod "c619c67a-04e0-45a2-8759-4f278b5b3326" (UID: "c619c67a-04e0-45a2-8759-4f278b5b3326"). InnerVolumeSpecName "kube-api-access-mnv4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:34.401892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:34.401851    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c619c67a-04e0-45a2-8759-4f278b5b3326-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:34.401892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:34.401883    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnv4h\" (UniqueName: \"kubernetes.io/projected/c619c67a-04e0-45a2-8759-4f278b5b3326-kube-api-access-mnv4h\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:35.196819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:35.196756    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z" event={"ID":"c619c67a-04e0-45a2-8759-4f278b5b3326","Type":"ContainerDied","Data":"6ae7d5f9e14345f85b98cefda5a8fe42169f1086dc6d3178e962318937bd8549"}
Nov 02 14:54:35.196819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:35.196809    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"
Nov 02 14:54:35.197358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:35.196847    5034 scope.go:122] "RemoveContainer" containerID="109e2d75d713964eb4bb00d30d4ac07cf9e4608c9a27a3798e2ecfacdc5dcf05"
Nov 02 14:54:35.255641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:35.255615    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"]
Nov 02 14:54:35.268909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:35.268880    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f30173-787b497d9c-kl42z"]
Nov 02 14:54:35.396142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:35.396113    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c619c67a-04e0-45a2-8759-4f278b5b3326" path="/var/lib/kubelet/pods/c619c67a-04e0-45a2-8759-4f278b5b3326/volumes"
Nov 02 14:54:39.537033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:39.536999    5034 scope.go:122] "RemoveContainer" containerID="ed864a37e84c60a7c1b5ce887f7b976c6cccfa3037a6ce3e615c3a0bf648d1fe"
Nov 02 14:54:39.546975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:39.546945    5034 scope.go:122] "RemoveContainer" containerID="65a26b7e13ca08ad1ba4cc9f8a9f6f20f64f1cf782da31991e580e1184b46b9c"
Nov 02 14:54:42.719490 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:42.719399    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"]
Nov 02 14:54:42.721228 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:42.720024    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy" containerID="cri-o://7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f" gracePeriod=30
Nov 02 14:54:48.028190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.028121    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 14:54:48.198664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.198626    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-credential-socket\") pod \"93258802-5287-409e-b9fb-71d0848c71c9\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") "
Nov 02 14:54:48.198868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.198715    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-socket\") pod \"93258802-5287-409e-b9fb-71d0848c71c9\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") "
Nov 02 14:54:48.198868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.198749    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-istio-token\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-istio-token\") pod \"93258802-5287-409e-b9fb-71d0848c71c9\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") "
Nov 02 14:54:48.198868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.198846    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-kube-api-access-8vjqn\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-kube-api-access-8vjqn\") pod \"93258802-5287-409e-b9fb-71d0848c71c9\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") "
Nov 02 14:54:48.199049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.198956    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93258802-5287-409e-b9fb-71d0848c71c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93258802-5287-409e-b9fb-71d0848c71c9-istiod-ca-cert\") pod \"93258802-5287-409e-b9fb-71d0848c71c9\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") "
Nov 02 14:54:48.199049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199000    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-credential-socket" pod "93258802-5287-409e-b9fb-71d0848c71c9" (UID: "93258802-5287-409e-b9fb-71d0848c71c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:48.199187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199051    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93258802-5287-409e-b9fb-71d0848c71c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93258802-5287-409e-b9fb-71d0848c71c9-istio-podinfo\") pod \"93258802-5287-409e-b9fb-71d0848c71c9\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") "
Nov 02 14:54:48.199187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199071    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-socket" pod "93258802-5287-409e-b9fb-71d0848c71c9" (UID: "93258802-5287-409e-b9fb-71d0848c71c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:48.199187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199086    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-certs\") pod \"93258802-5287-409e-b9fb-71d0848c71c9\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") "
Nov 02 14:54:48.199187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199169    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-data\") pod \"93258802-5287-409e-b9fb-71d0848c71c9\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") "
Nov 02 14:54:48.199417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199326    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-envoy\") pod \"93258802-5287-409e-b9fb-71d0848c71c9\" (UID: \"93258802-5287-409e-b9fb-71d0848c71c9\") "
Nov 02 14:54:48.199417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199355    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93258802-5287-409e-b9fb-71d0848c71c9-istiod-ca-cert" pod "93258802-5287-409e-b9fb-71d0848c71c9" (UID: "93258802-5287-409e-b9fb-71d0848c71c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:54:48.199583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199557    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-certs" pod "93258802-5287-409e-b9fb-71d0848c71c9" (UID: "93258802-5287-409e-b9fb-71d0848c71c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:48.199677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199602    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-data" pod "93258802-5287-409e-b9fb-71d0848c71c9" (UID: "93258802-5287-409e-b9fb-71d0848c71c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:48.199968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199947    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93258802-5287-409e-b9fb-71d0848c71c9-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:48.200111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199970    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:48.200111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199985    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:48.200111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.199998    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:48.200111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.200012    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:48.201521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.201490    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-kube-api-access-8vjqn" pod "93258802-5287-409e-b9fb-71d0848c71c9" (UID: "93258802-5287-409e-b9fb-71d0848c71c9"). InnerVolumeSpecName "kube-api-access-8vjqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:48.201667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.201633    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93258802-5287-409e-b9fb-71d0848c71c9-istio-podinfo" pod "93258802-5287-409e-b9fb-71d0848c71c9" (UID: "93258802-5287-409e-b9fb-71d0848c71c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:54:48.202012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.201992    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-istio-token" pod "93258802-5287-409e-b9fb-71d0848c71c9" (UID: "93258802-5287-409e-b9fb-71d0848c71c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:54:48.202086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.202010    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-envoy" pod "93258802-5287-409e-b9fb-71d0848c71c9" (UID: "93258802-5287-409e-b9fb-71d0848c71c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:54:48.302118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.302032    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93258802-5287-409e-b9fb-71d0848c71c9-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:48.302118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.302070    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93258802-5287-409e-b9fb-71d0848c71c9-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:48.302118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.302083    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:48.302118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.302099    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjqn\" (UniqueName: \"kubernetes.io/projected/93258802-5287-409e-b9fb-71d0848c71c9-kube-api-access-8vjqn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:54:48.309657 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.309622    5034 generic.go:356] "Generic (PLEG): container finished" podID="93258802-5287-409e-b9fb-71d0848c71c9" containerID="7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f" exitCode=0
Nov 02 14:54:48.309852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.309765    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" event={"ID":"93258802-5287-409e-b9fb-71d0848c71c9","Type":"ContainerDied","Data":"7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f"}
Nov 02 14:54:48.309852 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.309830    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5" event={"ID":"93258802-5287-409e-b9fb-71d0848c71c9","Type":"ContainerDied","Data":"1989584727c7608970cf24c2346988bf95e2f6f81b45bd36cc9dab90e8620575"}
Nov 02 14:54:48.309950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.309861    5034 scope.go:122] "RemoveContainer" containerID="7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f"
Nov 02 14:54:48.310057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.310033    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"
Nov 02 14:54:48.325087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.324884    5034 scope.go:122] "RemoveContainer" containerID="7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f"
Nov 02 14:54:48.325272 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:54:48.325205    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f\": container with ID starting with 7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f not found: ID does not exist" containerID="7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f"
Nov 02 14:54:48.325272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.325241    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f"} err="failed to get container status \"7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f\": rpc error: code = NotFound desc = could not find container \"7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f\": container with ID starting with 7e6e3a745b40bb9da390644ebfedbbf9545333a1fae58721e780d2d737ef5a1f not found: ID does not exist"
Nov 02 14:54:48.368254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.368223    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"]
Nov 02 14:54:48.372582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:48.372559    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f30173-openshift-default-7d49c94fc5-wsgl5"]
Nov 02 14:54:48.414944 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:54:48.414908    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:54:49.399869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:54:49.399835    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93258802-5287-409e-b9fb-71d0848c71c9" path="/var/lib/kubelet/pods/93258802-5287-409e-b9fb-71d0848c71c9/volumes"
Nov 02 14:55:18.415392 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:55:18.415354    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:55:39.603352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:39.603311    5034 scope.go:122] "RemoveContainer" containerID="808ba1e7ede9ff0a38e3d64dde3106a78d239dd88e3486c9c90b7b7b54b0bad9"
Nov 02 14:55:45.813118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.813086    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"]
Nov 02 14:55:45.815047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.813805    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c619c67a-04e0-45a2-8759-4f278b5b3326" containerName="payload-processing"
Nov 02 14:55:45.815047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.813820    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c619c67a-04e0-45a2-8759-4f278b5b3326" containerName="payload-processing"
Nov 02 14:55:45.815047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.813850    5034 cpu_manager.go:414] "containerMap: removing container" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy"
Nov 02 14:55:45.815047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.813856    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy"
Nov 02 14:55:45.815047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.813961    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c619c67a-04e0-45a2-8759-4f278b5b3326" containerName="payload-processing"
Nov 02 14:55:45.815047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.813975    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="93258802-5287-409e-b9fb-71d0848c71c9" containerName="istio-proxy"
Nov 02 14:55:45.817502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.817487    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:45.836878 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.836854    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"]
Nov 02 14:55:45.847838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.845413    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:55:45.847838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.845428    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:55:45.847838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.845473    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:55:45.847838 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.847348    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:55:45.914860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.914832    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ebde65-zhbgt\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:45.915012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.914899    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ebde65-zhbgt\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:45.915012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:45.914991    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ltlj\" (UniqueName: \"kubernetes.io/projected/98c8386b-255e-4bea-bdef-17ef12b17d23-kube-api-access-9ltlj\") pod \"maas-api-revoke-sub-b5ebde65-zhbgt\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:46.016439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.016385    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ebde65-zhbgt\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:46.016636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.016492    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ebde65-zhbgt\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:46.016636 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.016529    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ltlj\" (UniqueName: \"kubernetes.io/projected/98c8386b-255e-4bea-bdef-17ef12b17d23-kube-api-access-9ltlj\") pod \"maas-api-revoke-sub-b5ebde65-zhbgt\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:46.017199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.017173    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ebde65-zhbgt\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:46.018969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.018945    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ebde65-zhbgt\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:46.041903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.041883    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ltlj\" (UniqueName: \"kubernetes.io/projected/98c8386b-255e-4bea-bdef-17ef12b17d23-kube-api-access-9ltlj\") pod \"maas-api-revoke-sub-b5ebde65-zhbgt\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:46.135256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.135195    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:46.258033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.257995    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"]
Nov 02 14:55:46.263049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.263018    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.285206 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.285172    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"]
Nov 02 14:55:46.320142 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.320109    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f066e1-6z6q9\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.320310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.320148    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42qbm\" (UniqueName: \"kubernetes.io/projected/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-kube-api-access-42qbm\") pod \"maas-api-revoke-sub-47f066e1-6z6q9\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.320386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.320346    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f066e1-6z6q9\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.421645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.421568    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f066e1-6z6q9\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.421645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.421608    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42qbm\" (UniqueName: \"kubernetes.io/projected/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-kube-api-access-42qbm\") pod \"maas-api-revoke-sub-47f066e1-6z6q9\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.421953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.421807    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f066e1-6z6q9\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.422503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.422476    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47f066e1-6z6q9\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.424229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.424200    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47f066e1-6z6q9\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.442939 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:55:46.442908    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c8386b_255e_4bea_bdef_17ef12b17d23.slice/crio-a218772d26d9f5259ec920dcb509919e097f73b54a73e21dff93d96fe73ffa2a WatchSource:0}: Error finding container a218772d26d9f5259ec920dcb509919e097f73b54a73e21dff93d96fe73ffa2a: Status 404 returned error can't find the container with id a218772d26d9f5259ec920dcb509919e097f73b54a73e21dff93d96fe73ffa2a
Nov 02 14:55:46.444548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.444524    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"]
Nov 02 14:55:46.451836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.450966    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42qbm\" (UniqueName: \"kubernetes.io/projected/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-kube-api-access-42qbm\") pod \"maas-api-revoke-sub-47f066e1-6z6q9\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.579933 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.579905    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:46.808722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.808686    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt" event={"ID":"98c8386b-255e-4bea-bdef-17ef12b17d23","Type":"ContainerStarted","Data":"a218772d26d9f5259ec920dcb509919e097f73b54a73e21dff93d96fe73ffa2a"}
Nov 02 14:55:46.887540 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:55:46.887516    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6453ba6f_29bc_4c77_bd61_7bfdbdd54c7c.slice/crio-e1732a7f12aaac90f051af4e18f2f621dd73e03d7f873ad96543d7034771c833 WatchSource:0}: Error finding container e1732a7f12aaac90f051af4e18f2f621dd73e03d7f873ad96543d7034771c833: Status 404 returned error can't find the container with id e1732a7f12aaac90f051af4e18f2f621dd73e03d7f873ad96543d7034771c833
Nov 02 14:55:46.890854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:46.890825    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"]
Nov 02 14:55:47.821380 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:47.821351    5034 generic.go:356] "Generic (PLEG): container finished" podID="6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c" containerID="d3377fde76f56f0a345fdae6b69f7376a38e1515bd61fcb55ceefcdefae045df" exitCode=0
Nov 02 14:55:47.821545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:47.821435    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9" event={"ID":"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c","Type":"ContainerDied","Data":"d3377fde76f56f0a345fdae6b69f7376a38e1515bd61fcb55ceefcdefae045df"}
Nov 02 14:55:47.821545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:47.821489    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9" event={"ID":"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c","Type":"ContainerStarted","Data":"e1732a7f12aaac90f051af4e18f2f621dd73e03d7f873ad96543d7034771c833"}
Nov 02 14:55:47.823418 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:47.823390    5034 generic.go:356] "Generic (PLEG): container finished" podID="98c8386b-255e-4bea-bdef-17ef12b17d23" containerID="bd2740fd61d82b5af9d2c3a11dffc0debdc79f8e023f987a38ab9867c5603fd3" exitCode=0
Nov 02 14:55:47.823556 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:47.823469    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt" event={"ID":"98c8386b-255e-4bea-bdef-17ef12b17d23","Type":"ContainerDied","Data":"bd2740fd61d82b5af9d2c3a11dffc0debdc79f8e023f987a38ab9867c5603fd3"}
Nov 02 14:55:48.413615 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:55:48.413577    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:55:49.030883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.030850    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:49.060479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.060452    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:49.155955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.155871    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-xks-ca\") pod \"98c8386b-255e-4bea-bdef-17ef12b17d23\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") "
Nov 02 14:55:49.155955 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.155926    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-openshift-ca\") pod \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") "
Nov 02 14:55:49.156205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.155986    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-openshift-ca\") pod \"98c8386b-255e-4bea-bdef-17ef12b17d23\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") "
Nov 02 14:55:49.156205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.156098    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-kube-api-access-42qbm\" (UniqueName: \"kubernetes.io/projected/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-kube-api-access-42qbm\") pod \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") "
Nov 02 14:55:49.156328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.156221    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-xks-ca\") pod \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\" (UID: \"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c\") "
Nov 02 14:55:49.156328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.156271    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c8386b-255e-4bea-bdef-17ef12b17d23-kube-api-access-9ltlj\" (UniqueName: \"kubernetes.io/projected/98c8386b-255e-4bea-bdef-17ef12b17d23-kube-api-access-9ltlj\") pod \"98c8386b-255e-4bea-bdef-17ef12b17d23\" (UID: \"98c8386b-255e-4bea-bdef-17ef12b17d23\") "
Nov 02 14:55:49.156450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.156365    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-openshift-ca" pod "6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c" (UID: "6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:55:49.156676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.156652    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-openshift-ca" pod "98c8386b-255e-4bea-bdef-17ef12b17d23" (UID: "98c8386b-255e-4bea-bdef-17ef12b17d23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:55:49.157030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.157005    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:55:49.157135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.157036    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:55:49.159215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.158648    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-xks-ca" pod "98c8386b-255e-4bea-bdef-17ef12b17d23" (UID: "98c8386b-255e-4bea-bdef-17ef12b17d23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:55:49.159583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.159551    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c8386b-255e-4bea-bdef-17ef12b17d23-kube-api-access-9ltlj" pod "98c8386b-255e-4bea-bdef-17ef12b17d23" (UID: "98c8386b-255e-4bea-bdef-17ef12b17d23"). InnerVolumeSpecName "kube-api-access-9ltlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:55:49.163503 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.163455    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-kube-api-access-42qbm" pod "6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c" (UID: "6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c"). InnerVolumeSpecName "kube-api-access-42qbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:55:49.163615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.163495    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-xks-ca" pod "6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c" (UID: "6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:55:49.258638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.258612    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:55:49.258638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.258638    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ltlj\" (UniqueName: \"kubernetes.io/projected/98c8386b-255e-4bea-bdef-17ef12b17d23-kube-api-access-9ltlj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:55:49.258819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.258649    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98c8386b-255e-4bea-bdef-17ef12b17d23-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:55:49.258819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.258657    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42qbm\" (UniqueName: \"kubernetes.io/projected/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c-kube-api-access-42qbm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:55:49.841048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.841013    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"
Nov 02 14:55:49.841550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.841047    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9" event={"ID":"6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c","Type":"ContainerDied","Data":"e1732a7f12aaac90f051af4e18f2f621dd73e03d7f873ad96543d7034771c833"}
Nov 02 14:55:49.841550 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.841104    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1732a7f12aaac90f051af4e18f2f621dd73e03d7f873ad96543d7034771c833"
Nov 02 14:55:49.842708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.842691    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"
Nov 02 14:55:49.842708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.842693    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt" event={"ID":"98c8386b-255e-4bea-bdef-17ef12b17d23","Type":"ContainerDied","Data":"a218772d26d9f5259ec920dcb509919e097f73b54a73e21dff93d96fe73ffa2a"}
Nov 02 14:55:49.842911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:49.842744    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a218772d26d9f5259ec920dcb509919e097f73b54a73e21dff93d96fe73ffa2a"
Nov 02 14:55:56.186346 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.186313    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"]
Nov 02 14:55:56.187562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.187537    5034 cpu_manager.go:414] "containerMap: removing container" podUID="98c8386b-255e-4bea-bdef-17ef12b17d23" containerName="revoke-keys"
Nov 02 14:55:56.187562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.187563    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c8386b-255e-4bea-bdef-17ef12b17d23" containerName="revoke-keys"
Nov 02 14:55:56.187736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.187578    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c" containerName="revoke-keys"
Nov 02 14:55:56.187736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.187585    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c" containerName="revoke-keys"
Nov 02 14:55:56.187898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.187839    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c8386b-255e-4bea-bdef-17ef12b17d23" containerName="revoke-keys"
Nov 02 14:55:56.187898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.187857    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c" containerName="revoke-keys"
Nov 02 14:55:56.216554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.216514    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.224192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.222940    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"]
Nov 02 14:55:56.234036 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.233982    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:55:56.247644 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.247614    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:55:56.248121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.247618    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:55:56.248289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.247718    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:55:56.352210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.352176    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f920e209-318d-45df-a17a-c97ee585197f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76cf25a4-5xghd\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.352362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.352296    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f920e209-318d-45df-a17a-c97ee585197f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76cf25a4-5xghd\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.352424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.352388    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvchg\" (UniqueName: \"kubernetes.io/projected/f920e209-318d-45df-a17a-c97ee585197f-kube-api-access-tvchg\") pod \"maas-api-revoke-sub-76cf25a4-5xghd\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.453511 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.453432    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvchg\" (UniqueName: \"kubernetes.io/projected/f920e209-318d-45df-a17a-c97ee585197f-kube-api-access-tvchg\") pod \"maas-api-revoke-sub-76cf25a4-5xghd\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.453673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.453547    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f920e209-318d-45df-a17a-c97ee585197f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76cf25a4-5xghd\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.453673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.453647    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f920e209-318d-45df-a17a-c97ee585197f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76cf25a4-5xghd\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.454407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.454382    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f920e209-318d-45df-a17a-c97ee585197f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76cf25a4-5xghd\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.456082 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.456061    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f920e209-318d-45df-a17a-c97ee585197f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76cf25a4-5xghd\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.478627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.478592    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvchg\" (UniqueName: \"kubernetes.io/projected/f920e209-318d-45df-a17a-c97ee585197f-kube-api-access-tvchg\") pod \"maas-api-revoke-sub-76cf25a4-5xghd\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.538075 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.538046    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:56.573824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.573791    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"]
Nov 02 14:55:56.579687 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.579663    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.603487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.601076    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"]
Nov 02 14:55:56.657001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.656971    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7bf411e-9plfn\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.657149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.657027    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7bf411e-9plfn\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.657149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.657112    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7rv\" (UniqueName: \"kubernetes.io/projected/556008cb-1991-4c8a-9030-d06d48b3dfd7-kube-api-access-pm7rv\") pod \"maas-api-revoke-sub-f7bf411e-9plfn\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.758605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.758568    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7bf411e-9plfn\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.758770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.758631    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7bf411e-9plfn\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.758770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.758735    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7rv\" (UniqueName: \"kubernetes.io/projected/556008cb-1991-4c8a-9030-d06d48b3dfd7-kube-api-access-pm7rv\") pod \"maas-api-revoke-sub-f7bf411e-9plfn\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.759276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.759257    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7bf411e-9plfn\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.761145 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.761120    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7bf411e-9plfn\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.784150 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.784121    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7rv\" (UniqueName: \"kubernetes.io/projected/556008cb-1991-4c8a-9030-d06d48b3dfd7-kube-api-access-pm7rv\") pod \"maas-api-revoke-sub-f7bf411e-9plfn\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.862906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.862879    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"]
Nov 02 14:55:56.906561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.906530    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:55:56.909928 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:56.909898    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd" event={"ID":"f920e209-318d-45df-a17a-c97ee585197f","Type":"ContainerStarted","Data":"c61dc39c9dc7ea9a83c76ce4ea0e798690c8f34efa0da39cb468ff48396f67da"}
Nov 02 14:55:57.220512 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:55:57.220486    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556008cb_1991_4c8a_9030_d06d48b3dfd7.slice/crio-0adbef74a6df0b59bb649c48e299afcdc740da39b23249df702a08ac63247021 WatchSource:0}: Error finding container 0adbef74a6df0b59bb649c48e299afcdc740da39b23249df702a08ac63247021: Status 404 returned error can't find the container with id 0adbef74a6df0b59bb649c48e299afcdc740da39b23249df702a08ac63247021
Nov 02 14:55:57.226926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:57.226901    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"]
Nov 02 14:55:57.921565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:57.921527    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn" event={"ID":"556008cb-1991-4c8a-9030-d06d48b3dfd7","Type":"ContainerStarted","Data":"4d10b4534952d985dfe743db8e4dcac674507dedadd9d4dded05c46a459aaed2"}
Nov 02 14:55:57.921737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:57.921593    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn" event={"ID":"556008cb-1991-4c8a-9030-d06d48b3dfd7","Type":"ContainerStarted","Data":"0adbef74a6df0b59bb649c48e299afcdc740da39b23249df702a08ac63247021"}
Nov 02 14:55:57.923531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:57.923500    5034 generic.go:356] "Generic (PLEG): container finished" podID="f920e209-318d-45df-a17a-c97ee585197f" containerID="996db8c6c6e98c61862a95b22a17c2aed54bdcd4dc8738bd2554f5eedfd4be07" exitCode=0
Nov 02 14:55:57.923661 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:57.923582    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd" event={"ID":"f920e209-318d-45df-a17a-c97ee585197f","Type":"ContainerDied","Data":"996db8c6c6e98c61862a95b22a17c2aed54bdcd4dc8738bd2554f5eedfd4be07"}
Nov 02 14:55:57.974266 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:57.974192    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn" podStartSLOduration=1.974171004 podStartE2EDuration="1.974171004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:55:57.964958125 +0000 UTC m=+4348.716262563" watchObservedRunningTime="2026-11-02 14:55:57.974171004 +0000 UTC m=+4348.725475444"
Nov 02 14:55:58.934931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:58.934882    5034 generic.go:356] "Generic (PLEG): container finished" podID="556008cb-1991-4c8a-9030-d06d48b3dfd7" containerID="4d10b4534952d985dfe743db8e4dcac674507dedadd9d4dded05c46a459aaed2" exitCode=0
Nov 02 14:55:58.935375 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:58.934944    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn" event={"ID":"556008cb-1991-4c8a-9030-d06d48b3dfd7","Type":"ContainerDied","Data":"4d10b4534952d985dfe743db8e4dcac674507dedadd9d4dded05c46a459aaed2"}
Nov 02 14:55:59.259015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.246336    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:55:59.317703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.312271    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f920e209-318d-45df-a17a-c97ee585197f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f920e209-318d-45df-a17a-c97ee585197f-maas-api-openshift-ca\") pod \"f920e209-318d-45df-a17a-c97ee585197f\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") "
Nov 02 14:55:59.317703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.312407    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f920e209-318d-45df-a17a-c97ee585197f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f920e209-318d-45df-a17a-c97ee585197f-maas-api-xks-ca\") pod \"f920e209-318d-45df-a17a-c97ee585197f\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") "
Nov 02 14:55:59.317703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.312547    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f920e209-318d-45df-a17a-c97ee585197f-kube-api-access-tvchg\" (UniqueName: \"kubernetes.io/projected/f920e209-318d-45df-a17a-c97ee585197f-kube-api-access-tvchg\") pod \"f920e209-318d-45df-a17a-c97ee585197f\" (UID: \"f920e209-318d-45df-a17a-c97ee585197f\") "
Nov 02 14:55:59.333799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.328345    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f920e209-318d-45df-a17a-c97ee585197f-maas-api-openshift-ca" pod "f920e209-318d-45df-a17a-c97ee585197f" (UID: "f920e209-318d-45df-a17a-c97ee585197f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:55:59.359808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.356321    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f920e209-318d-45df-a17a-c97ee585197f-maas-api-xks-ca" pod "f920e209-318d-45df-a17a-c97ee585197f" (UID: "f920e209-318d-45df-a17a-c97ee585197f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:55:59.369804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.364985    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f920e209-318d-45df-a17a-c97ee585197f-kube-api-access-tvchg" pod "f920e209-318d-45df-a17a-c97ee585197f" (UID: "f920e209-318d-45df-a17a-c97ee585197f"). InnerVolumeSpecName "kube-api-access-tvchg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:55:59.416796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.415200    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f920e209-318d-45df-a17a-c97ee585197f-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:55:59.416796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.415237    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvchg\" (UniqueName: \"kubernetes.io/projected/f920e209-318d-45df-a17a-c97ee585197f-kube-api-access-tvchg\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:55:59.416796 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.415253    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f920e209-318d-45df-a17a-c97ee585197f-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:00.003804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:55:59.999537    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"
Nov 02 14:56:00.003804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.000259    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd" event={"ID":"f920e209-318d-45df-a17a-c97ee585197f","Type":"ContainerDied","Data":"c61dc39c9dc7ea9a83c76ce4ea0e798690c8f34efa0da39cb468ff48396f67da"}
Nov 02 14:56:00.003804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.000309    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c61dc39c9dc7ea9a83c76ce4ea0e798690c8f34efa0da39cb468ff48396f67da"
Nov 02 14:56:00.295081 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.294953    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:56:00.439797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.439112    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556008cb-1991-4c8a-9030-d06d48b3dfd7-kube-api-access-pm7rv\" (UniqueName: \"kubernetes.io/projected/556008cb-1991-4c8a-9030-d06d48b3dfd7-kube-api-access-pm7rv\") pod \"556008cb-1991-4c8a-9030-d06d48b3dfd7\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") "
Nov 02 14:56:00.439797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.439242    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-xks-ca\") pod \"556008cb-1991-4c8a-9030-d06d48b3dfd7\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") "
Nov 02 14:56:00.439797 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.439325    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-openshift-ca\") pod \"556008cb-1991-4c8a-9030-d06d48b3dfd7\" (UID: \"556008cb-1991-4c8a-9030-d06d48b3dfd7\") "
Nov 02 14:56:00.440096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.439961    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-openshift-ca" pod "556008cb-1991-4c8a-9030-d06d48b3dfd7" (UID: "556008cb-1991-4c8a-9030-d06d48b3dfd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:00.440620 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.440592    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:00.441711 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.441677    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-xks-ca" pod "556008cb-1991-4c8a-9030-d06d48b3dfd7" (UID: "556008cb-1991-4c8a-9030-d06d48b3dfd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:00.442730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.442698    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556008cb-1991-4c8a-9030-d06d48b3dfd7-kube-api-access-pm7rv" pod "556008cb-1991-4c8a-9030-d06d48b3dfd7" (UID: "556008cb-1991-4c8a-9030-d06d48b3dfd7"). InnerVolumeSpecName "kube-api-access-pm7rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:00.542302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.542264    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm7rv\" (UniqueName: \"kubernetes.io/projected/556008cb-1991-4c8a-9030-d06d48b3dfd7-kube-api-access-pm7rv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:00.542302 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:00.542303    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/556008cb-1991-4c8a-9030-d06d48b3dfd7-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:01.019216 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:01.019187    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"
Nov 02 14:56:01.019709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:01.019229    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn" event={"ID":"556008cb-1991-4c8a-9030-d06d48b3dfd7","Type":"ContainerDied","Data":"0adbef74a6df0b59bb649c48e299afcdc740da39b23249df702a08ac63247021"}
Nov 02 14:56:01.019709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:01.019280    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0adbef74a6df0b59bb649c48e299afcdc740da39b23249df702a08ac63247021"
Nov 02 14:56:06.037605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:06.037572    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"]
Nov 02 14:56:06.050953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:06.050922    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d608aec-h62gs"]
Nov 02 14:56:07.396605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:07.396572    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf2886c-a17e-4255-a8be-5790ddacd3fd" path="/var/lib/kubelet/pods/3bf2886c-a17e-4255-a8be-5790ddacd3fd/volumes"
Nov 02 14:56:09.177448 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.177415    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"]
Nov 02 14:56:09.178095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.178079    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f920e209-318d-45df-a17a-c97ee585197f" containerName="revoke-keys"
Nov 02 14:56:09.178095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.178094    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f920e209-318d-45df-a17a-c97ee585197f" containerName="revoke-keys"
Nov 02 14:56:09.178232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.178135    5034 cpu_manager.go:414] "containerMap: removing container" podUID="556008cb-1991-4c8a-9030-d06d48b3dfd7" containerName="revoke-keys"
Nov 02 14:56:09.178232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.178141    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556008cb-1991-4c8a-9030-d06d48b3dfd7" containerName="revoke-keys"
Nov 02 14:56:09.178352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.178263    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f920e209-318d-45df-a17a-c97ee585197f" containerName="revoke-keys"
Nov 02 14:56:09.178352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.178280    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="556008cb-1991-4c8a-9030-d06d48b3dfd7" containerName="revoke-keys"
Nov 02 14:56:09.181860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.181839    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.203057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.203024    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"]
Nov 02 14:56:09.211250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.211202    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:56:09.212083 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.212058    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:56:09.212420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.212096    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:56:09.212420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.212135    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:56:09.258419 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.258392    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsl7s\" (UniqueName: \"kubernetes.io/projected/79598a5a-d187-43dd-9415-795f2d9867cd-kube-api-access-bsl7s\") pod \"maas-api-revoke-sub-1ca422c6-s7cjv\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.258557 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.258449    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca422c6-s7cjv\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.258624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.258604    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca422c6-s7cjv\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.360001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.359967    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca422c6-s7cjv\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.360182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.360124    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsl7s\" (UniqueName: \"kubernetes.io/projected/79598a5a-d187-43dd-9415-795f2d9867cd-kube-api-access-bsl7s\") pod \"maas-api-revoke-sub-1ca422c6-s7cjv\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.360255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.360180    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca422c6-s7cjv\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.361022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.360992    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ca422c6-s7cjv\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.362482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.362454    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ca422c6-s7cjv\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.385647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.385613    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsl7s\" (UniqueName: \"kubernetes.io/projected/79598a5a-d187-43dd-9415-795f2d9867cd-kube-api-access-bsl7s\") pod \"maas-api-revoke-sub-1ca422c6-s7cjv\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.496623 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.496598    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:09.804310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:09.804278    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"]
Nov 02 14:56:09.805026 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:56:09.804996    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79598a5a_d187_43dd_9415_795f2d9867cd.slice/crio-c7e1c0d3309e7080c09ede1313853e700815136743874c24b6e5c88b06ce6644 WatchSource:0}: Error finding container c7e1c0d3309e7080c09ede1313853e700815136743874c24b6e5c88b06ce6644: Status 404 returned error can't find the container with id c7e1c0d3309e7080c09ede1313853e700815136743874c24b6e5c88b06ce6644
Nov 02 14:56:10.096829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:10.096703    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv" event={"ID":"79598a5a-d187-43dd-9415-795f2d9867cd","Type":"ContainerStarted","Data":"c7e1c0d3309e7080c09ede1313853e700815136743874c24b6e5c88b06ce6644"}
Nov 02 14:56:11.108617 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:11.108582    5034 generic.go:356] "Generic (PLEG): container finished" podID="79598a5a-d187-43dd-9415-795f2d9867cd" containerID="cdd80374fda154145b2a53c4f361666741b5f1475d69f7762384ca75aae1150a" exitCode=0
Nov 02 14:56:11.109159 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:11.108664    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv" event={"ID":"79598a5a-d187-43dd-9415-795f2d9867cd","Type":"ContainerDied","Data":"cdd80374fda154145b2a53c4f361666741b5f1475d69f7762384ca75aae1150a"}
Nov 02 14:56:12.304333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.304304    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:12.402291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.402206    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79598a5a-d187-43dd-9415-795f2d9867cd-kube-api-access-bsl7s\" (UniqueName: \"kubernetes.io/projected/79598a5a-d187-43dd-9415-795f2d9867cd-kube-api-access-bsl7s\") pod \"79598a5a-d187-43dd-9415-795f2d9867cd\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") "
Nov 02 14:56:12.402467 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.402410    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-openshift-ca\") pod \"79598a5a-d187-43dd-9415-795f2d9867cd\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") "
Nov 02 14:56:12.402549 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.402471    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-xks-ca\") pod \"79598a5a-d187-43dd-9415-795f2d9867cd\" (UID: \"79598a5a-d187-43dd-9415-795f2d9867cd\") "
Nov 02 14:56:12.402882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.402847    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-openshift-ca" pod "79598a5a-d187-43dd-9415-795f2d9867cd" (UID: "79598a5a-d187-43dd-9415-795f2d9867cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:12.403066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.402988    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:12.405242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.405185    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-xks-ca" pod "79598a5a-d187-43dd-9415-795f2d9867cd" (UID: "79598a5a-d187-43dd-9415-795f2d9867cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:12.405331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.405236    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79598a5a-d187-43dd-9415-795f2d9867cd-kube-api-access-bsl7s" pod "79598a5a-d187-43dd-9415-795f2d9867cd" (UID: "79598a5a-d187-43dd-9415-795f2d9867cd"). InnerVolumeSpecName "kube-api-access-bsl7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:12.504923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.504887    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79598a5a-d187-43dd-9415-795f2d9867cd-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:12.504923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:12.504928    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsl7s\" (UniqueName: \"kubernetes.io/projected/79598a5a-d187-43dd-9415-795f2d9867cd-kube-api-access-bsl7s\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:13.127242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:13.127214    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"
Nov 02 14:56:13.127433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:13.127257    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv" event={"ID":"79598a5a-d187-43dd-9415-795f2d9867cd","Type":"ContainerDied","Data":"c7e1c0d3309e7080c09ede1313853e700815136743874c24b6e5c88b06ce6644"}
Nov 02 14:56:13.127433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:13.127313    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e1c0d3309e7080c09ede1313853e700815136743874c24b6e5c88b06ce6644"
Nov 02 14:56:18.415260 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:56:18.415220    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:56:19.516911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.516881    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"]
Nov 02 14:56:19.517757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.517731    5034 cpu_manager.go:414] "containerMap: removing container" podUID="79598a5a-d187-43dd-9415-795f2d9867cd" containerName="revoke-keys"
Nov 02 14:56:19.517757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.517757    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79598a5a-d187-43dd-9415-795f2d9867cd" containerName="revoke-keys"
Nov 02 14:56:19.518023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.518007    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="79598a5a-d187-43dd-9415-795f2d9867cd" containerName="revoke-keys"
Nov 02 14:56:19.521907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.521886    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.539250 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.539229    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:56:19.539368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.539352    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:56:19.539433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.539420    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:56:19.540011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.539994    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"]
Nov 02 14:56:19.548868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.548850    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:56:19.617121 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.617085    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e75454d-bmwrp\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.617291 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.617271    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8nkt\" (UniqueName: \"kubernetes.io/projected/c921b24f-8d1a-471e-98c9-0c6ab4418857-kube-api-access-w8nkt\") pod \"maas-api-revoke-sub-1e75454d-bmwrp\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.617437 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.617414    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e75454d-bmwrp\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.718697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.718666    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8nkt\" (UniqueName: \"kubernetes.io/projected/c921b24f-8d1a-471e-98c9-0c6ab4418857-kube-api-access-w8nkt\") pod \"maas-api-revoke-sub-1e75454d-bmwrp\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.718846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.718743    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e75454d-bmwrp\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.718846 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.718827    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e75454d-bmwrp\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.719381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.719356    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e75454d-bmwrp\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.721223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.721198    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e75454d-bmwrp\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.743328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.743306    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8nkt\" (UniqueName: \"kubernetes.io/projected/c921b24f-8d1a-471e-98c9-0c6ab4418857-kube-api-access-w8nkt\") pod \"maas-api-revoke-sub-1e75454d-bmwrp\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:19.835729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:19.835670    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:20.145003 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:56:20.144970    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc921b24f_8d1a_471e_98c9_0c6ab4418857.slice/crio-c2693634524023d283619e8b2ddbd0f1762fdcdfe04cff309175e24b30ca3019 WatchSource:0}: Error finding container c2693634524023d283619e8b2ddbd0f1762fdcdfe04cff309175e24b30ca3019: Status 404 returned error can't find the container with id c2693634524023d283619e8b2ddbd0f1762fdcdfe04cff309175e24b30ca3019
Nov 02 14:56:20.145947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:20.145920    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"]
Nov 02 14:56:20.207668 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:20.207637    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp" event={"ID":"c921b24f-8d1a-471e-98c9-0c6ab4418857","Type":"ContainerStarted","Data":"c2693634524023d283619e8b2ddbd0f1762fdcdfe04cff309175e24b30ca3019"}
Nov 02 14:56:21.217888 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:21.217849    5034 generic.go:356] "Generic (PLEG): container finished" podID="c921b24f-8d1a-471e-98c9-0c6ab4418857" containerID="8f2167360a83f7554f4ee38e672c229e4087e6b2633f7a68d4abdf71a075814d" exitCode=0
Nov 02 14:56:21.218379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:21.217933    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp" event={"ID":"c921b24f-8d1a-471e-98c9-0c6ab4418857","Type":"ContainerDied","Data":"8f2167360a83f7554f4ee38e672c229e4087e6b2633f7a68d4abdf71a075814d"}
Nov 02 14:56:21.492096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:21.492062    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"]
Nov 02 14:56:22.403500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.403474    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:22.569819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.569787    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c921b24f-8d1a-471e-98c9-0c6ab4418857-kube-api-access-w8nkt\" (UniqueName: \"kubernetes.io/projected/c921b24f-8d1a-471e-98c9-0c6ab4418857-kube-api-access-w8nkt\") pod \"c921b24f-8d1a-471e-98c9-0c6ab4418857\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") "
Nov 02 14:56:22.569997 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.569929    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-openshift-ca\") pod \"c921b24f-8d1a-471e-98c9-0c6ab4418857\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") "
Nov 02 14:56:22.570096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.570082    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-xks-ca\") pod \"c921b24f-8d1a-471e-98c9-0c6ab4418857\" (UID: \"c921b24f-8d1a-471e-98c9-0c6ab4418857\") "
Nov 02 14:56:22.570282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.570255    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-openshift-ca" pod "c921b24f-8d1a-471e-98c9-0c6ab4418857" (UID: "c921b24f-8d1a-471e-98c9-0c6ab4418857"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:22.570568 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.570544    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:22.572001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.571974    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-xks-ca" pod "c921b24f-8d1a-471e-98c9-0c6ab4418857" (UID: "c921b24f-8d1a-471e-98c9-0c6ab4418857"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:22.572297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.572281    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c921b24f-8d1a-471e-98c9-0c6ab4418857-kube-api-access-w8nkt" pod "c921b24f-8d1a-471e-98c9-0c6ab4418857" (UID: "c921b24f-8d1a-471e-98c9-0c6ab4418857"). InnerVolumeSpecName "kube-api-access-w8nkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:22.671627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.671596    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c921b24f-8d1a-471e-98c9-0c6ab4418857-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:22.671627 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:22.671625    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8nkt\" (UniqueName: \"kubernetes.io/projected/c921b24f-8d1a-471e-98c9-0c6ab4418857-kube-api-access-w8nkt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:23.236326 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:23.236297    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"
Nov 02 14:56:23.236482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:23.236330    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp" event={"ID":"c921b24f-8d1a-471e-98c9-0c6ab4418857","Type":"ContainerDied","Data":"c2693634524023d283619e8b2ddbd0f1762fdcdfe04cff309175e24b30ca3019"}
Nov 02 14:56:23.236482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:23.236381    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2693634524023d283619e8b2ddbd0f1762fdcdfe04cff309175e24b30ca3019"
Nov 02 14:56:25.025869 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:25.025833    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"]
Nov 02 14:56:25.059930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:25.059894    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"]
Nov 02 14:56:26.493311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:26.493273    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh" podUID="0604c248-3e5a-4de8-8772-932e884a4326" containerName="payload-pre-processing" containerID="cri-o://613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f" gracePeriod=25
Nov 02 14:56:30.027106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.027067    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf" podUID="3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" containerName="payload-processing" containerID="cri-o://1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698" gracePeriod=25
Nov 02 14:56:30.060641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.060600    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr" podUID="30e257b7-094d-48da-a548-3ec863b5f148" containerName="payload-pre-processing" containerID="cri-o://295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39" gracePeriod=25
Nov 02 14:56:30.404763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.404681    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"]
Nov 02 14:56:30.405485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.405463    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c921b24f-8d1a-471e-98c9-0c6ab4418857" containerName="revoke-keys"
Nov 02 14:56:30.405485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.405487    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c921b24f-8d1a-471e-98c9-0c6ab4418857" containerName="revoke-keys"
Nov 02 14:56:30.405673 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.405657    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c921b24f-8d1a-471e-98c9-0c6ab4418857" containerName="revoke-keys"
Nov 02 14:56:30.409813 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.409794    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.426588 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.426567    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:56:30.429889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.429860    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"]
Nov 02 14:56:30.437677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.437653    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:56:30.437821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.437735    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:56:30.441007 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.440980    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:56:30.483496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.483467    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24611a13-n4qcq\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.483625 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.483515    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24611a13-n4qcq\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.483811 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.483762    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr2zp\" (UniqueName: \"kubernetes.io/projected/9bf3e04f-0208-4557-bb5d-39393d57ee04-kube-api-access-tr2zp\") pod \"maas-api-revoke-sub-24611a13-n4qcq\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.585055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.585019    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24611a13-n4qcq\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.585233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.585114    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24611a13-n4qcq\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.585339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.585321    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr2zp\" (UniqueName: \"kubernetes.io/projected/9bf3e04f-0208-4557-bb5d-39393d57ee04-kube-api-access-tr2zp\") pod \"maas-api-revoke-sub-24611a13-n4qcq\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.585788 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.585737    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24611a13-n4qcq\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.587727 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.587704    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24611a13-n4qcq\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.610277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.610247    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr2zp\" (UniqueName: \"kubernetes.io/projected/9bf3e04f-0208-4557-bb5d-39393d57ee04-kube-api-access-tr2zp\") pod \"maas-api-revoke-sub-24611a13-n4qcq\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.724700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.724634    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:30.862357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.862315    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf" podUID="3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:56:30.868038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:30.867996    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr" podUID="30e257b7-094d-48da-a548-3ec863b5f148" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:56:31.031203 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:56:31.031173    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf3e04f_0208_4557_bb5d_39393d57ee04.slice/crio-b991030974c15d466ce56ce9714a857683abb9fb27a6e1353c146165f8872673 WatchSource:0}: Error finding container b991030974c15d466ce56ce9714a857683abb9fb27a6e1353c146165f8872673: Status 404 returned error can't find the container with id b991030974c15d466ce56ce9714a857683abb9fb27a6e1353c146165f8872673
Nov 02 14:56:31.034269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:31.034243    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"]
Nov 02 14:56:31.300567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:31.300480    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq" event={"ID":"9bf3e04f-0208-4557-bb5d-39393d57ee04","Type":"ContainerStarted","Data":"b991030974c15d466ce56ce9714a857683abb9fb27a6e1353c146165f8872673"}
Nov 02 14:56:32.080124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:32.080077    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh" podUID="0604c248-3e5a-4de8-8772-932e884a4326" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:56:32.311978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:32.311941    5034 generic.go:356] "Generic (PLEG): container finished" podID="9bf3e04f-0208-4557-bb5d-39393d57ee04" containerID="a0e3befede25eedfba05dd551a0596f7571c8219940bc379fd48c2df34f13d9b" exitCode=0
Nov 02 14:56:32.312164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:32.312033    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq" event={"ID":"9bf3e04f-0208-4557-bb5d-39393d57ee04","Type":"ContainerDied","Data":"a0e3befede25eedfba05dd551a0596f7571c8219940bc379fd48c2df34f13d9b"}
Nov 02 14:56:33.489460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.489428    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:33.532531 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.532505    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-xks-ca\") pod \"9bf3e04f-0208-4557-bb5d-39393d57ee04\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") "
Nov 02 14:56:33.532684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.532546    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf3e04f-0208-4557-bb5d-39393d57ee04-kube-api-access-tr2zp\" (UniqueName: \"kubernetes.io/projected/9bf3e04f-0208-4557-bb5d-39393d57ee04-kube-api-access-tr2zp\") pod \"9bf3e04f-0208-4557-bb5d-39393d57ee04\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") "
Nov 02 14:56:33.532828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.532752    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-openshift-ca\") pod \"9bf3e04f-0208-4557-bb5d-39393d57ee04\" (UID: \"9bf3e04f-0208-4557-bb5d-39393d57ee04\") "
Nov 02 14:56:33.533160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.533106    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-openshift-ca" pod "9bf3e04f-0208-4557-bb5d-39393d57ee04" (UID: "9bf3e04f-0208-4557-bb5d-39393d57ee04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:33.533958 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.533688    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:33.538816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.536113    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf3e04f-0208-4557-bb5d-39393d57ee04-kube-api-access-tr2zp" pod "9bf3e04f-0208-4557-bb5d-39393d57ee04" (UID: "9bf3e04f-0208-4557-bb5d-39393d57ee04"). InnerVolumeSpecName "kube-api-access-tr2zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:33.538816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.536116    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-xks-ca" pod "9bf3e04f-0208-4557-bb5d-39393d57ee04" (UID: "9bf3e04f-0208-4557-bb5d-39393d57ee04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:33.635060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.634990    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf3e04f-0208-4557-bb5d-39393d57ee04-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:33.635060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:33.635020    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr2zp\" (UniqueName: \"kubernetes.io/projected/9bf3e04f-0208-4557-bb5d-39393d57ee04-kube-api-access-tr2zp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:34.331131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:34.331095    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"
Nov 02 14:56:34.331131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:34.331108    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq" event={"ID":"9bf3e04f-0208-4557-bb5d-39393d57ee04","Type":"ContainerDied","Data":"b991030974c15d466ce56ce9714a857683abb9fb27a6e1353c146165f8872673"}
Nov 02 14:56:34.331328 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:34.331164    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b991030974c15d466ce56ce9714a857683abb9fb27a6e1353c146165f8872673"
Nov 02 14:56:40.837479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:40.837435    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr" podUID="30e257b7-094d-48da-a548-3ec863b5f148" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:56:40.837934 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:40.837509    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf" podUID="3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:56:40.993593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:40.993548    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"]
Nov 02 14:56:40.994372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:40.994348    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9bf3e04f-0208-4557-bb5d-39393d57ee04" containerName="revoke-keys"
Nov 02 14:56:40.994372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:40.994371    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf3e04f-0208-4557-bb5d-39393d57ee04" containerName="revoke-keys"
Nov 02 14:56:40.994564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:40.994540    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf3e04f-0208-4557-bb5d-39393d57ee04" containerName="revoke-keys"
Nov 02 14:56:40.998496 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:40.998477    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.020275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.020244    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"]
Nov 02 14:56:41.028991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.028966    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:56:41.030205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.030151    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:56:41.030205 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.030198    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:56:41.030381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.030307    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:56:41.146368 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.146291    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fa19b7a-69dk7\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.146513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.146432    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fa19b7a-69dk7\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.146513 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.146496    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9qcq\" (UniqueName: \"kubernetes.io/projected/c0d60b57-6e94-4c11-9128-f9668c124b9f-kube-api-access-n9qcq\") pod \"maas-api-revoke-sub-5fa19b7a-69dk7\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.247245 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.247211    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fa19b7a-69dk7\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.247399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.247281    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9qcq\" (UniqueName: \"kubernetes.io/projected/c0d60b57-6e94-4c11-9128-f9668c124b9f-kube-api-access-n9qcq\") pod \"maas-api-revoke-sub-5fa19b7a-69dk7\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.247399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.247386    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fa19b7a-69dk7\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.248023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.248000    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fa19b7a-69dk7\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.249541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.249519    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fa19b7a-69dk7\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.272715 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.272683    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9qcq\" (UniqueName: \"kubernetes.io/projected/c0d60b57-6e94-4c11-9128-f9668c124b9f-kube-api-access-n9qcq\") pod \"maas-api-revoke-sub-5fa19b7a-69dk7\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.312880 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.312847    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:41.629650 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:56:41.629618    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d60b57_6e94_4c11_9128_f9668c124b9f.slice/crio-2806026b4d6a24783922593bce89c04cae675889ebbdbccc473fc58f81918979 WatchSource:0}: Error finding container 2806026b4d6a24783922593bce89c04cae675889ebbdbccc473fc58f81918979: Status 404 returned error can't find the container with id 2806026b4d6a24783922593bce89c04cae675889ebbdbccc473fc58f81918979
Nov 02 14:56:41.631330 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:41.631282    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"]
Nov 02 14:56:42.079854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:42.079815    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh" podUID="0604c248-3e5a-4de8-8772-932e884a4326" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:56:42.419186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:42.419154    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7" event={"ID":"c0d60b57-6e94-4c11-9128-f9668c124b9f","Type":"ContainerStarted","Data":"96bf5771ef3379f7f1c55989a095a2abff12749db51e2691c9d33886761a6413"}
Nov 02 14:56:42.419392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:42.419208    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7" event={"ID":"c0d60b57-6e94-4c11-9128-f9668c124b9f","Type":"ContainerStarted","Data":"2806026b4d6a24783922593bce89c04cae675889ebbdbccc473fc58f81918979"}
Nov 02 14:56:42.463143 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:42.463100    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7" podStartSLOduration=2.463087733 podStartE2EDuration="2.463087733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:42.462419395 +0000 UTC m=+4393.213723833" watchObservedRunningTime="2026-11-02 14:56:42.463087733 +0000 UTC m=+4393.214392170"
Nov 02 14:56:43.427336 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:43.427301    5034 generic.go:356] "Generic (PLEG): container finished" podID="c0d60b57-6e94-4c11-9128-f9668c124b9f" containerID="96bf5771ef3379f7f1c55989a095a2abff12749db51e2691c9d33886761a6413" exitCode=0
Nov 02 14:56:43.427807 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:43.427392    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7" event={"ID":"c0d60b57-6e94-4c11-9128-f9668c124b9f","Type":"ContainerDied","Data":"96bf5771ef3379f7f1c55989a095a2abff12749db51e2691c9d33886761a6413"}
Nov 02 14:56:44.612982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.612956    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:44.692804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.692709    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-xks-ca\") pod \"c0d60b57-6e94-4c11-9128-f9668c124b9f\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") "
Nov 02 14:56:44.692971 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.692870    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-openshift-ca\") pod \"c0d60b57-6e94-4c11-9128-f9668c124b9f\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") "
Nov 02 14:56:44.693049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.692987    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d60b57-6e94-4c11-9128-f9668c124b9f-kube-api-access-n9qcq\" (UniqueName: \"kubernetes.io/projected/c0d60b57-6e94-4c11-9128-f9668c124b9f-kube-api-access-n9qcq\") pod \"c0d60b57-6e94-4c11-9128-f9668c124b9f\" (UID: \"c0d60b57-6e94-4c11-9128-f9668c124b9f\") "
Nov 02 14:56:44.693561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.693527    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-openshift-ca" pod "c0d60b57-6e94-4c11-9128-f9668c124b9f" (UID: "c0d60b57-6e94-4c11-9128-f9668c124b9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:44.694100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.694072    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:44.708808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.695688    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-xks-ca" pod "c0d60b57-6e94-4c11-9128-f9668c124b9f" (UID: "c0d60b57-6e94-4c11-9128-f9668c124b9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:56:44.708808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.696016    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d60b57-6e94-4c11-9128-f9668c124b9f-kube-api-access-n9qcq" pod "c0d60b57-6e94-4c11-9128-f9668c124b9f" (UID: "c0d60b57-6e94-4c11-9128-f9668c124b9f"). InnerVolumeSpecName "kube-api-access-n9qcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:44.808899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.799943    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d60b57-6e94-4c11-9128-f9668c124b9f-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:44.808899 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:44.799984    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9qcq\" (UniqueName: \"kubernetes.io/projected/c0d60b57-6e94-4c11-9128-f9668c124b9f-kube-api-access-n9qcq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:45.444487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:45.444451    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7" event={"ID":"c0d60b57-6e94-4c11-9128-f9668c124b9f","Type":"ContainerDied","Data":"2806026b4d6a24783922593bce89c04cae675889ebbdbccc473fc58f81918979"}
Nov 02 14:56:45.444487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:45.444485    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"
Nov 02 14:56:45.444710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:45.444508    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2806026b4d6a24783922593bce89c04cae675889ebbdbccc473fc58f81918979"
Nov 02 14:56:46.776905 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:46.776872    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:56:46.928835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:46.928788    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0604c248-3e5a-4de8-8772-932e884a4326-kube-api-access-qkqlz\" (UniqueName: \"kubernetes.io/projected/0604c248-3e5a-4de8-8772-932e884a4326-kube-api-access-qkqlz\") pod \"0604c248-3e5a-4de8-8772-932e884a4326\" (UID: \"0604c248-3e5a-4de8-8772-932e884a4326\") "
Nov 02 14:56:46.929047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:46.928899    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0604c248-3e5a-4de8-8772-932e884a4326-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0604c248-3e5a-4de8-8772-932e884a4326-plugins-config-volume\") pod \"0604c248-3e5a-4de8-8772-932e884a4326\" (UID: \"0604c248-3e5a-4de8-8772-932e884a4326\") "
Nov 02 14:56:46.929353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:46.929326    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0604c248-3e5a-4de8-8772-932e884a4326-plugins-config-volume" pod "0604c248-3e5a-4de8-8772-932e884a4326" (UID: "0604c248-3e5a-4de8-8772-932e884a4326"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:46.931156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:46.931128    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0604c248-3e5a-4de8-8772-932e884a4326-kube-api-access-qkqlz" pod "0604c248-3e5a-4de8-8772-932e884a4326" (UID: "0604c248-3e5a-4de8-8772-932e884a4326"). InnerVolumeSpecName "kube-api-access-qkqlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:47.030663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.030588    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0604c248-3e5a-4de8-8772-932e884a4326-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:47.030663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.030614    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkqlz\" (UniqueName: \"kubernetes.io/projected/0604c248-3e5a-4de8-8772-932e884a4326-kube-api-access-qkqlz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:47.460420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.460388    5034 generic.go:356] "Generic (PLEG): container finished" podID="0604c248-3e5a-4de8-8772-932e884a4326" containerID="613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f" exitCode=0
Nov 02 14:56:47.460584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.460432    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"
Nov 02 14:56:47.460584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.460472    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh" event={"ID":"0604c248-3e5a-4de8-8772-932e884a4326","Type":"ContainerDied","Data":"613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f"}
Nov 02 14:56:47.460584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.460540    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh" event={"ID":"0604c248-3e5a-4de8-8772-932e884a4326","Type":"ContainerDied","Data":"579044597e7aaa86143fd809b3d9dce62d37de39f809cd8ef76e5beb40b06940"}
Nov 02 14:56:47.460584 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.460580    5034 scope.go:122] "RemoveContainer" containerID="613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f"
Nov 02 14:56:47.472735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.472715    5034 scope.go:122] "RemoveContainer" containerID="613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f"
Nov 02 14:56:47.473020 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:56:47.473002    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f\": container with ID starting with 613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f not found: ID does not exist" containerID="613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f"
Nov 02 14:56:47.473084 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.473029    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f"} err="failed to get container status \"613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f\": rpc error: code = NotFound desc = could not find container \"613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f\": container with ID starting with 613647460c151ffa0fc90000c344cbc0b209bf2170b0c5b076ed1bd92203c27f not found: ID does not exist"
Nov 02 14:56:47.524008 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.523981    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"]
Nov 02 14:56:47.537853 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:47.537830    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-26b508-5c6c7d7859-gd9xh"]
Nov 02 14:56:48.413953 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:56:48.413910    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:56:49.396257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:49.396226    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0604c248-3e5a-4de8-8772-932e884a4326" path="/var/lib/kubelet/pods/0604c248-3e5a-4de8-8772-932e884a4326/volumes"
Nov 02 14:56:50.419968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.419936    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:56:50.443860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.443836    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:56:50.539765 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.539730    5034 generic.go:356] "Generic (PLEG): container finished" podID="3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" containerID="1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698" exitCode=0
Nov 02 14:56:50.539990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.539817    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"
Nov 02 14:56:50.539990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.539817    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf" event={"ID":"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac","Type":"ContainerDied","Data":"1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698"}
Nov 02 14:56:50.540134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.540035    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf" event={"ID":"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac","Type":"ContainerDied","Data":"09c6838a34dbddf26a07bf3946e55cb211c4735a6160e69843dbd6b63f3c31a9"}
Nov 02 14:56:50.540134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.540078    5034 scope.go:122] "RemoveContainer" containerID="1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698"
Nov 02 14:56:50.542012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.541818    5034 generic.go:356] "Generic (PLEG): container finished" podID="30e257b7-094d-48da-a548-3ec863b5f148" containerID="295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39" exitCode=0
Nov 02 14:56:50.542012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.541860    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"
Nov 02 14:56:50.542012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.541909    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr" event={"ID":"30e257b7-094d-48da-a548-3ec863b5f148","Type":"ContainerDied","Data":"295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39"}
Nov 02 14:56:50.542012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.541959    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr" event={"ID":"30e257b7-094d-48da-a548-3ec863b5f148","Type":"ContainerDied","Data":"e4bc61ce425e679825cab773b37820549786884cc2928cb1b04cc3ba536f7bc8"}
Nov 02 14:56:50.553207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.553181    5034 scope.go:122] "RemoveContainer" containerID="1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698"
Nov 02 14:56:50.553509 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:56:50.553482    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698\": container with ID starting with 1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698 not found: ID does not exist" containerID="1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698"
Nov 02 14:56:50.553574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.553517    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698"} err="failed to get container status \"1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698\": rpc error: code = NotFound desc = could not find container \"1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698\": container with ID starting with 1296fa80178d6676374de193d01eea247dde844e2bc406a0a28a2f6ae62a4698 not found: ID does not exist"
Nov 02 14:56:50.553574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.553534    5034 scope.go:122] "RemoveContainer" containerID="295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39"
Nov 02 14:56:50.564077 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.564060    5034 scope.go:122] "RemoveContainer" containerID="295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39"
Nov 02 14:56:50.564341 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:56:50.564319    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39\": container with ID starting with 295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39 not found: ID does not exist" containerID="295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39"
Nov 02 14:56:50.564433 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.564350    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39"} err="failed to get container status \"295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39\": rpc error: code = NotFound desc = could not find container \"295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39\": container with ID starting with 295064c05dc4fd962306fb19edaadc4795df2f6aa132ad17b33d4a34274a3e39 not found: ID does not exist"
Nov 02 14:56:50.587153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.587103    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e257b7-094d-48da-a548-3ec863b5f148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e257b7-094d-48da-a548-3ec863b5f148-plugins-config-volume\") pod \"30e257b7-094d-48da-a548-3ec863b5f148\" (UID: \"30e257b7-094d-48da-a548-3ec863b5f148\") "
Nov 02 14:56:50.587153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.587148    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-kube-api-access-dvs75\" (UniqueName: \"kubernetes.io/projected/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-kube-api-access-dvs75\") pod \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\" (UID: \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\") "
Nov 02 14:56:50.587333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.587171    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e257b7-094d-48da-a548-3ec863b5f148-kube-api-access-dwbgb\" (UniqueName: \"kubernetes.io/projected/30e257b7-094d-48da-a548-3ec863b5f148-kube-api-access-dwbgb\") pod \"30e257b7-094d-48da-a548-3ec863b5f148\" (UID: \"30e257b7-094d-48da-a548-3ec863b5f148\") "
Nov 02 14:56:50.587333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.587242    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-plugins-config-volume\") pod \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\" (UID: \"3805ee4d-ff1c-47f8-bd49-00b7b21e4eac\") "
Nov 02 14:56:50.587529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.587509    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e257b7-094d-48da-a548-3ec863b5f148-plugins-config-volume" pod "30e257b7-094d-48da-a548-3ec863b5f148" (UID: "30e257b7-094d-48da-a548-3ec863b5f148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:50.587664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.587643    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-plugins-config-volume" pod "3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" (UID: "3805ee4d-ff1c-47f8-bd49-00b7b21e4eac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:56:50.587855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.587837    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:50.587924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.587858    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e257b7-094d-48da-a548-3ec863b5f148-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:50.589292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.589271    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-kube-api-access-dvs75" pod "3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" (UID: "3805ee4d-ff1c-47f8-bd49-00b7b21e4eac"). InnerVolumeSpecName "kube-api-access-dvs75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:50.589389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.589295    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e257b7-094d-48da-a548-3ec863b5f148-kube-api-access-dwbgb" pod "30e257b7-094d-48da-a548-3ec863b5f148" (UID: "30e257b7-094d-48da-a548-3ec863b5f148"). InnerVolumeSpecName "kube-api-access-dwbgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:56:50.689068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.689042    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvs75\" (UniqueName: \"kubernetes.io/projected/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac-kube-api-access-dvs75\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:50.689068 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.689068    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwbgb\" (UniqueName: \"kubernetes.io/projected/30e257b7-094d-48da-a548-3ec863b5f148-kube-api-access-dwbgb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:56:50.899112 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.899075    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"]
Nov 02 14:56:50.912308 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.912262    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ce0f55-6d5cdf6bf5-8rwqr"]
Nov 02 14:56:50.947157 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.947126    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"]
Nov 02 14:56:50.951694 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:50.951670    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ce0f55-6fc6fc798c-k5lwf"]
Nov 02 14:56:51.400913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:51.400876    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e257b7-094d-48da-a548-3ec863b5f148" path="/var/lib/kubelet/pods/30e257b7-094d-48da-a548-3ec863b5f148/volumes"
Nov 02 14:56:51.401478 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:51.401458    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" path="/var/lib/kubelet/pods/3805ee4d-ff1c-47f8-bd49-00b7b21e4eac/volumes"
Nov 02 14:56:55.477405 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:55.477367    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"]
Nov 02 14:56:55.500109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:56:55.500080    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"]
Nov 02 14:57:00.478443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:00.478393    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp" podUID="fd7597dc-d18f-46e0-b4df-6242defcbbe6" containerName="payload-processing" containerID="cri-o://284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3" gracePeriod=25
Nov 02 14:57:00.501058 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:00.501018    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv" podUID="c830edf9-caad-47fd-88aa-1c1e24e50790" containerName="payload-pre-processing" containerID="cri-o://f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24" gracePeriod=25
Nov 02 14:57:00.978298 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:00.978254    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"]
Nov 02 14:57:01.012502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:01.012474    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"]
Nov 02 14:57:01.114913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:01.114878    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp" podUID="fd7597dc-d18f-46e0-b4df-6242defcbbe6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:05.857725 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:05.857685    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"]
Nov 02 14:57:05.858259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:05.857962    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy" containerID="cri-o://71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011" gracePeriod=30
Nov 02 14:57:05.983721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:05.983029    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs" podUID="5b6c78f1-4137-4146-958b-726d999c64b5" containerName="payload-processing" containerID="cri-o://96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b" gracePeriod=25
Nov 02 14:57:06.013624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:06.013584    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz" podUID="4ac001bf-2ca9-4400-a244-f4404bf2351c" containerName="payload-pre-processing" containerID="cri-o://c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c" gracePeriod=25
Nov 02 14:57:07.463155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:07.463121    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs" podUID="5b6c78f1-4137-4146-958b-726d999c64b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:07.463581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:07.463130    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz" podUID="4ac001bf-2ca9-4400-a244-f4404bf2351c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:07.818652 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:07.818615    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"]
Nov 02 14:57:07.819585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:07.819546    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy" containerID="cri-o://8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c" gracePeriod=30
Nov 02 14:57:10.099546 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:10.099506    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv" podUID="c830edf9-caad-47fd-88aa-1c1e24e50790" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:11.114746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.114697    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp" podUID="fd7597dc-d18f-46e0-b4df-6242defcbbe6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:11.161907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.161873    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:57:11.302350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302313    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-socket\") pod \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") "
Nov 02 14:57:11.302563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302387    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-podinfo\") pod \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") "
Nov 02 14:57:11.302563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302494    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-token\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-token\") pod \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") "
Nov 02 14:57:11.302707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302598    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-kube-api-access-gf48q\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-kube-api-access-gf48q\") pod \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") "
Nov 02 14:57:11.302707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302624    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-socket" pod "55f5b619-f69a-45ff-904a-d8507d2c2ea6" (UID: "55f5b619-f69a-45ff-904a-d8507d2c2ea6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:11.302707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302664    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-certs\") pod \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") "
Nov 02 14:57:11.302707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302705    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-credential-socket\") pod \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") "
Nov 02 14:57:11.302964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302799    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istiod-ca-cert\") pod \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") "
Nov 02 14:57:11.302964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302837    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-envoy\") pod \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") "
Nov 02 14:57:11.302964 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302866    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-data\") pod \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\" (UID: \"55f5b619-f69a-45ff-904a-d8507d2c2ea6\") "
Nov 02 14:57:11.303134 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.302989    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-certs" pod "55f5b619-f69a-45ff-904a-d8507d2c2ea6" (UID: "55f5b619-f69a-45ff-904a-d8507d2c2ea6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:11.303508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.303244    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-credential-socket" pod "55f5b619-f69a-45ff-904a-d8507d2c2ea6" (UID: "55f5b619-f69a-45ff-904a-d8507d2c2ea6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:11.303508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.303465    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-data" pod "55f5b619-f69a-45ff-904a-d8507d2c2ea6" (UID: "55f5b619-f69a-45ff-904a-d8507d2c2ea6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:11.303697 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.303567    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istiod-ca-cert" pod "55f5b619-f69a-45ff-904a-d8507d2c2ea6" (UID: "55f5b619-f69a-45ff-904a-d8507d2c2ea6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:11.303760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.303734    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:11.303848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.303756    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:11.303848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.303774    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:11.303848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.303807    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:11.303848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.303819    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:11.305069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.305038    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-kube-api-access-gf48q" pod "55f5b619-f69a-45ff-904a-d8507d2c2ea6" (UID: "55f5b619-f69a-45ff-904a-d8507d2c2ea6"). InnerVolumeSpecName "kube-api-access-gf48q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:11.305197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.305068    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-podinfo" pod "55f5b619-f69a-45ff-904a-d8507d2c2ea6" (UID: "55f5b619-f69a-45ff-904a-d8507d2c2ea6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:11.305382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.305355    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-envoy" pod "55f5b619-f69a-45ff-904a-d8507d2c2ea6" (UID: "55f5b619-f69a-45ff-904a-d8507d2c2ea6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:11.305978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.305955    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-token" pod "55f5b619-f69a-45ff-904a-d8507d2c2ea6" (UID: "55f5b619-f69a-45ff-904a-d8507d2c2ea6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:11.404643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.404563    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:11.404643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.404589    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:11.404643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.404599    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:11.404643 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.404607    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf48q\" (UniqueName: \"kubernetes.io/projected/55f5b619-f69a-45ff-904a-d8507d2c2ea6-kube-api-access-gf48q\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:11.761674 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.761642    5034 generic.go:356] "Generic (PLEG): container finished" podID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerID="71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011" exitCode=0
Nov 02 14:57:11.761912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.761689    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"
Nov 02 14:57:11.761912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.761722    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" event={"ID":"55f5b619-f69a-45ff-904a-d8507d2c2ea6","Type":"ContainerDied","Data":"71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011"}
Nov 02 14:57:11.761912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.761791    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm" event={"ID":"55f5b619-f69a-45ff-904a-d8507d2c2ea6","Type":"ContainerDied","Data":"d15c53a7885cb840dbf2200f57242435e802592f401c8fc70e49f86560ee37d7"}
Nov 02 14:57:11.761912 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.761822    5034 scope.go:122] "RemoveContainer" containerID="71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011"
Nov 02 14:57:11.772950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.772928    5034 scope.go:122] "RemoveContainer" containerID="d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589"
Nov 02 14:57:11.783232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.783211    5034 scope.go:122] "RemoveContainer" containerID="71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011"
Nov 02 14:57:11.783562 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:11.783536    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011\": container with ID starting with 71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011 not found: ID does not exist" containerID="71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011"
Nov 02 14:57:11.783634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.783575    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011"} err="failed to get container status \"71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011\": rpc error: code = NotFound desc = could not find container \"71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011\": container with ID starting with 71c22b62915eeb73e863fdcba50a8db2d1f59ae054ba50b89bc7c54bedbf4011 not found: ID does not exist"
Nov 02 14:57:11.783634 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.783602    5034 scope.go:122] "RemoveContainer" containerID="d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589"
Nov 02 14:57:11.783947 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:11.783926    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589\": container with ID starting with d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589 not found: ID does not exist" containerID="d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589"
Nov 02 14:57:11.784023 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.783951    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589"} err="failed to get container status \"d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589\": rpc error: code = NotFound desc = could not find container \"d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589\": container with ID starting with d0df39a50b9b6b84f6fd0e11ff8acb583999388c3ff7d43a5e11a4c1c1d12589 not found: ID does not exist"
Nov 02 14:57:11.820616 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.820591    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"]
Nov 02 14:57:11.834545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:11.834516    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-66bc6d-openshift-default-7747656cbf-55hcm"]
Nov 02 14:57:12.026017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:12.025936    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"]
Nov 02 14:57:12.026197 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:12.026171    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy" containerID="cri-o://60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57" gracePeriod=30
Nov 02 14:57:12.030100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:12.030076    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"]
Nov 02 14:57:12.030301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:12.030279    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy" containerID="cri-o://ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb" gracePeriod=30
Nov 02 14:57:12.033501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:12.033478    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"]
Nov 02 14:57:12.033693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:12.033672    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy" containerID="cri-o://8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6" gracePeriod=30
Nov 02 14:57:12.036950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:12.036926    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"]
Nov 02 14:57:12.037101 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:12.037085    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy" containerID="cri-o://3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166" gracePeriod=30
Nov 02 14:57:13.128537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.128511    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:57:13.230439 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.230404    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-podinfo\") pod \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") "
Nov 02 14:57:13.230607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.230495    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-kube-api-access-m25wv\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-kube-api-access-m25wv\") pod \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") "
Nov 02 14:57:13.230607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.230561    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-token\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-token\") pod \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") "
Nov 02 14:57:13.230734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.230617    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istiod-ca-cert\") pod \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") "
Nov 02 14:57:13.230734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.230654    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-socket\") pod \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") "
Nov 02 14:57:13.230734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.230704    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-credential-socket\") pod \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") "
Nov 02 14:57:13.230918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.230762    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-data\") pod \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") "
Nov 02 14:57:13.230985 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.230921    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-certs\") pod \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") "
Nov 02 14:57:13.230985 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.230944    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-envoy\") pod \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\" (UID: \"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9\") "
Nov 02 14:57:13.231819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.231213    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-credential-socket" pod "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" (UID: "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:13.231819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.231520    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-socket" pod "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" (UID: "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:13.231819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.231729    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-certs" pod "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" (UID: "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:13.231819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.231762    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:13.231819 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.231805    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:13.232265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.232011    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-data" pod "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" (UID: "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:13.232265 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.232149    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istiod-ca-cert" pod "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" (UID: "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:13.233311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.233267    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-podinfo" pod "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" (UID: "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:13.234046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.233852    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-kube-api-access-m25wv" pod "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" (UID: "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9"). InnerVolumeSpecName "kube-api-access-m25wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:13.234046 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.234010    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-envoy" pod "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" (UID: "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:13.234276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.234252    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-token" pod "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" (UID: "1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:13.333738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.333708    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:13.333738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.333737    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:13.333738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.333745    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:13.334079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.333754    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m25wv\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-kube-api-access-m25wv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:13.334079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.333766    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:13.334079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.333810    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:13.334079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.333824    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:13.396325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.396244    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" path="/var/lib/kubelet/pods/55f5b619-f69a-45ff-904a-d8507d2c2ea6/volumes"
Nov 02 14:57:13.785659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.785611    5034 generic.go:356] "Generic (PLEG): container finished" podID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerID="8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c" exitCode=0
Nov 02 14:57:13.785914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.785889    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" event={"ID":"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9","Type":"ContainerDied","Data":"8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c"}
Nov 02 14:57:13.785999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.785950    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9" event={"ID":"1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9","Type":"ContainerDied","Data":"7e8799b8be951a34bacf4a8f7c8189faa136f1c712b0e19037877432e1759171"}
Nov 02 14:57:13.785999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.785983    5034 scope.go:122] "RemoveContainer" containerID="8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c"
Nov 02 14:57:13.786133 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.786112    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"
Nov 02 14:57:13.814146 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.813804    5034 scope.go:122] "RemoveContainer" containerID="f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60"
Nov 02 14:57:13.829335 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.828873    5034 scope.go:122] "RemoveContainer" containerID="8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c"
Nov 02 14:57:13.829911 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:13.829888    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c\": container with ID starting with 8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c not found: ID does not exist" containerID="8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c"
Nov 02 14:57:13.830045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.829921    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c"} err="failed to get container status \"8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c\": rpc error: code = NotFound desc = could not find container \"8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c\": container with ID starting with 8016cb1d5549170cbe05fec72093534abefcc2d1e0dc8632ef90a09f76a7d98c not found: ID does not exist"
Nov 02 14:57:13.830045 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.829944    5034 scope.go:122] "RemoveContainer" containerID="f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60"
Nov 02 14:57:13.830279 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:13.830249    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60\": container with ID starting with f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60 not found: ID does not exist" containerID="f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60"
Nov 02 14:57:13.830456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.830285    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60"} err="failed to get container status \"f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60\": rpc error: code = NotFound desc = could not find container \"f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60\": container with ID starting with f8bc07baea73cca70ab8ae183e97dc61dbb009137edc018532e08ce53f889b60 not found: ID does not exist"
Nov 02 14:57:13.864292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.864264    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"]
Nov 02 14:57:13.877403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:13.877379    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-5f98a6-openshift-default-5854557845-zt8t9"]
Nov 02 14:57:14.864578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:14.864547    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"]
Nov 02 14:57:15.396210 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:15.396169    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" path="/var/lib/kubelet/pods/1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9/volumes"
Nov 02 14:57:17.384799 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.362315    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:57:17.417747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.417094    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-socket\") pod \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") "
Nov 02 14:57:17.417747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.417140    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istiod-ca-cert\") pod \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") "
Nov 02 14:57:17.417747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.417263    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-token\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-token\") pod \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") "
Nov 02 14:57:17.417747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.417300    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-certs\") pod \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") "
Nov 02 14:57:17.417747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.417342    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-kube-api-access-ldcg6\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-kube-api-access-ldcg6\") pod \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") "
Nov 02 14:57:17.417747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.417396    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-credential-socket\") pod \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") "
Nov 02 14:57:17.417747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.417464    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-data\") pod \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") "
Nov 02 14:57:17.417747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.417495    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-envoy\") pod \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") "
Nov 02 14:57:17.417747 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.417651    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-podinfo\") pod \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\" (UID: \"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49\") "
Nov 02 14:57:17.418452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.418023    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-credential-socket" pod "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" (UID: "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.418452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.418034    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-certs" pod "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" (UID: "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.418452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.418224    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-data" pod "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" (UID: "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.418452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.418322    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istiod-ca-cert" pod "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" (UID: "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:17.418452 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.418379    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-socket" pod "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" (UID: "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.420817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.419394    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.420817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.419421    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.420817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.419438    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.420817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.419452    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.420817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.419468    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.423155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.423128    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-envoy" pod "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" (UID: "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.423304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.423170    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-kube-api-access-ldcg6" pod "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" (UID: "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49"). InnerVolumeSpecName "kube-api-access-ldcg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.423841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.423642    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-token" pod "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" (UID: "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.424859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.424830    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-podinfo" pod "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" (UID: "59ea47e2-abbe-43d5-9225-3ce3b7c0dc49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:17.440837 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.440815    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:57:17.444192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.444169    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:57:17.448929 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.448905    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:57:17.463540 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.463508    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs" podUID="5b6c78f1-4137-4146-958b-726d999c64b5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:17.463656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.463572    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz" podUID="4ac001bf-2ca9-4400-a244-f4404bf2351c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:17.520484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520451    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-socket\") pod \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") "
Nov 02 14:57:17.520484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520490    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-credential-socket\") pod \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") "
Nov 02 14:57:17.520736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520515    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-token\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-token\") pod \"e1666e4c-a00a-408b-8fd5-4570510a96c4\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") "
Nov 02 14:57:17.520736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520547    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-certs\") pod \"e1666e4c-a00a-408b-8fd5-4570510a96c4\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") "
Nov 02 14:57:17.520736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520588    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-socket\") pod \"e1666e4c-a00a-408b-8fd5-4570510a96c4\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") "
Nov 02 14:57:17.520736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520627    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-envoy\") pod \"0bdca348-be51-47a6-bf14-32e7c181f160\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") "
Nov 02 14:57:17.520736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520656    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-data\") pod \"e1666e4c-a00a-408b-8fd5-4570510a96c4\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") "
Nov 02 14:57:17.520736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520708    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-data\") pod \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") "
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520770    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-data\") pod \"0bdca348-be51-47a6-bf14-32e7c181f160\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") "
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520808    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-socket" pod "76c5fc52-0dad-47ac-ab36-f9ad186eabf8" (UID: "76c5fc52-0dad-47ac-ab36-f9ad186eabf8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520834    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-token\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-token\") pod \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") "
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520850    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-credential-socket" pod "76c5fc52-0dad-47ac-ab36-f9ad186eabf8" (UID: "76c5fc52-0dad-47ac-ab36-f9ad186eabf8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520883    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-podinfo\") pod \"e1666e4c-a00a-408b-8fd5-4570510a96c4\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") "
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520919    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bdca348-be51-47a6-bf14-32e7c181f160-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdca348-be51-47a6-bf14-32e7c181f160-istiod-ca-cert\") pod \"0bdca348-be51-47a6-bf14-32e7c181f160\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") "
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.520946    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-kube-api-access-nwwv4\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-kube-api-access-nwwv4\") pod \"e1666e4c-a00a-408b-8fd5-4570510a96c4\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") "
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521034    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bdca348-be51-47a6-bf14-32e7c181f160-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdca348-be51-47a6-bf14-32e7c181f160-istio-podinfo\") pod \"0bdca348-be51-47a6-bf14-32e7c181f160\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") "
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521061    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-kube-api-access-dmlp4\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-kube-api-access-dmlp4\") pod \"0bdca348-be51-47a6-bf14-32e7c181f160\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") "
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521067    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-certs" pod "e1666e4c-a00a-408b-8fd5-4570510a96c4" (UID: "e1666e4c-a00a-408b-8fd5-4570510a96c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521070    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-data" pod "76c5fc52-0dad-47ac-ab36-f9ad186eabf8" (UID: "76c5fc52-0dad-47ac-ab36-f9ad186eabf8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521084    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-socket" pod "e1666e4c-a00a-408b-8fd5-4570510a96c4" (UID: "e1666e4c-a00a-408b-8fd5-4570510a96c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.521100 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521093    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-certs\") pod \"0bdca348-be51-47a6-bf14-32e7c181f160\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") "
Nov 02 14:57:17.521841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521250    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-data" pod "e1666e4c-a00a-408b-8fd5-4570510a96c4" (UID: "e1666e4c-a00a-408b-8fd5-4570510a96c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.521841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521326    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-certs" pod "0bdca348-be51-47a6-bf14-32e7c181f160" (UID: "0bdca348-be51-47a6-bf14-32e7c181f160"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.521841 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521521    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdca348-be51-47a6-bf14-32e7c181f160-istiod-ca-cert" pod "0bdca348-be51-47a6-bf14-32e7c181f160" (UID: "0bdca348-be51-47a6-bf14-32e7c181f160"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:17.522012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.521960    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-socket\") pod \"0bdca348-be51-47a6-bf14-32e7c181f160\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") "
Nov 02 14:57:17.522092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522070    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-credential-socket\") pod \"0bdca348-be51-47a6-bf14-32e7c181f160\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") "
Nov 02 14:57:17.522148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522108    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-podinfo\") pod \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") "
Nov 02 14:57:17.522213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522159    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-kube-api-access-vsnjz\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-kube-api-access-vsnjz\") pod \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") "
Nov 02 14:57:17.522213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522181    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-data" pod "0bdca348-be51-47a6-bf14-32e7c181f160" (UID: "0bdca348-be51-47a6-bf14-32e7c181f160"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.522331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522211    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-envoy\") pod \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") "
Nov 02 14:57:17.522331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522238    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-certs\") pod \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") "
Nov 02 14:57:17.522331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522269    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-istio-token\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-istio-token\") pod \"0bdca348-be51-47a6-bf14-32e7c181f160\" (UID: \"0bdca348-be51-47a6-bf14-32e7c181f160\") "
Nov 02 14:57:17.522331 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522302    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-envoy\") pod \"e1666e4c-a00a-408b-8fd5-4570510a96c4\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") "
Nov 02 14:57:17.522567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522336    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-socket" pod "0bdca348-be51-47a6-bf14-32e7c181f160" (UID: "0bdca348-be51-47a6-bf14-32e7c181f160"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.522567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522382    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-credential-socket\") pod \"e1666e4c-a00a-408b-8fd5-4570510a96c4\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") "
Nov 02 14:57:17.522567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522427    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-credential-socket" pod "0bdca348-be51-47a6-bf14-32e7c181f160" (UID: "0bdca348-be51-47a6-bf14-32e7c181f160"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.522567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522430    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1666e4c-a00a-408b-8fd5-4570510a96c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1666e4c-a00a-408b-8fd5-4570510a96c4-istiod-ca-cert\") pod \"e1666e4c-a00a-408b-8fd5-4570510a96c4\" (UID: \"e1666e4c-a00a-408b-8fd5-4570510a96c4\") "
Nov 02 14:57:17.522567 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522487    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istiod-ca-cert\") pod \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\" (UID: \"76c5fc52-0dad-47ac-ab36-f9ad186eabf8\") "
Nov 02 14:57:17.522882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522715    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1666e4c-a00a-408b-8fd5-4570510a96c4-istiod-ca-cert" pod "e1666e4c-a00a-408b-8fd5-4570510a96c4" (UID: "e1666e4c-a00a-408b-8fd5-4570510a96c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:17.522969 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.522948    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-certs" pod "76c5fc52-0dad-47ac-ab36-f9ad186eabf8" (UID: "76c5fc52-0dad-47ac-ab36-f9ad186eabf8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.523225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523205    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523225    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istiod-ca-cert" pod "76c5fc52-0dad-47ac-ab36-f9ad186eabf8" (UID: "76c5fc52-0dad-47ac-ab36-f9ad186eabf8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:17.523310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523230    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523256    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523270    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdca348-be51-47a6-bf14-32e7c181f160-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523285    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523299    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523313    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523329    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523389    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-credential-socket" pod "e1666e4c-a00a-408b-8fd5-4570510a96c4" (UID: "e1666e4c-a00a-408b-8fd5-4570510a96c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523431    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523449    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1666e4c-a00a-408b-8fd5-4570510a96c4-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523463    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523476    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523491    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523504    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523515    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523527    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldcg6\" (UniqueName: \"kubernetes.io/projected/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49-kube-api-access-ldcg6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.523689 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.523541    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.526062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.526038    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-podinfo" pod "76c5fc52-0dad-47ac-ab36-f9ad186eabf8" (UID: "76c5fc52-0dad-47ac-ab36-f9ad186eabf8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:17.526195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.526089    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-envoy" pod "e1666e4c-a00a-408b-8fd5-4570510a96c4" (UID: "e1666e4c-a00a-408b-8fd5-4570510a96c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.526195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.526123    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-envoy" pod "76c5fc52-0dad-47ac-ab36-f9ad186eabf8" (UID: "76c5fc52-0dad-47ac-ab36-f9ad186eabf8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.526407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.526385    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-kube-api-access-dmlp4" pod "0bdca348-be51-47a6-bf14-32e7c181f160" (UID: "0bdca348-be51-47a6-bf14-32e7c181f160"). InnerVolumeSpecName "kube-api-access-dmlp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.526514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.526422    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-token" pod "76c5fc52-0dad-47ac-ab36-f9ad186eabf8" (UID: "76c5fc52-0dad-47ac-ab36-f9ad186eabf8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.526514 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.526503    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-podinfo" pod "e1666e4c-a00a-408b-8fd5-4570510a96c4" (UID: "e1666e4c-a00a-408b-8fd5-4570510a96c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:17.526683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.526553    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-kube-api-access-nwwv4" pod "e1666e4c-a00a-408b-8fd5-4570510a96c4" (UID: "e1666e4c-a00a-408b-8fd5-4570510a96c4"). InnerVolumeSpecName "kube-api-access-nwwv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.527289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.527260    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-token" pod "e1666e4c-a00a-408b-8fd5-4570510a96c4" (UID: "e1666e4c-a00a-408b-8fd5-4570510a96c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.527386 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.527368    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-kube-api-access-vsnjz" pod "76c5fc52-0dad-47ac-ab36-f9ad186eabf8" (UID: "76c5fc52-0dad-47ac-ab36-f9ad186eabf8"). InnerVolumeSpecName "kube-api-access-vsnjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.527735 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.527713    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bdca348-be51-47a6-bf14-32e7c181f160-istio-podinfo" pod "0bdca348-be51-47a6-bf14-32e7c181f160" (UID: "0bdca348-be51-47a6-bf14-32e7c181f160"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:17.527883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.527868    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-istio-token" pod "0bdca348-be51-47a6-bf14-32e7c181f160" (UID: "0bdca348-be51-47a6-bf14-32e7c181f160"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:17.528756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.528738    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-envoy" pod "0bdca348-be51-47a6-bf14-32e7c181f160" (UID: "0bdca348-be51-47a6-bf14-32e7c181f160"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:17.625624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625546    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625584    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdca348-be51-47a6-bf14-32e7c181f160-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625597    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625611    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625624 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625624    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwv4\" (UniqueName: \"kubernetes.io/projected/e1666e4c-a00a-408b-8fd5-4570510a96c4-kube-api-access-nwwv4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625640    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdca348-be51-47a6-bf14-32e7c181f160-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625654    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmlp4\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-kube-api-access-dmlp4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625669    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625681    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsnjz\" (UniqueName: \"kubernetes.io/projected/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-kube-api-access-vsnjz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625695    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625707    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdca348-be51-47a6-bf14-32e7c181f160-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625718    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625730    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1666e4c-a00a-408b-8fd5-4570510a96c4-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.625961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.625741    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76c5fc52-0dad-47ac-ab36-f9ad186eabf8-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:17.824646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.824605    5034 generic.go:356] "Generic (PLEG): container finished" podID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerID="8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6" exitCode=0
Nov 02 14:57:17.824866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.824666    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"
Nov 02 14:57:17.824866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.824688    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" event={"ID":"e1666e4c-a00a-408b-8fd5-4570510a96c4","Type":"ContainerDied","Data":"8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6"}
Nov 02 14:57:17.824866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.824754    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9" event={"ID":"e1666e4c-a00a-408b-8fd5-4570510a96c4","Type":"ContainerDied","Data":"578f0f9ce46dd50c43df6c846965b713925e51c004a1f8a82d1cd5f59424b65c"}
Nov 02 14:57:17.824866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.824816    5034 scope.go:122] "RemoveContainer" containerID="8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6"
Nov 02 14:57:17.826501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.826476    5034 generic.go:356] "Generic (PLEG): container finished" podID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerID="60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57" exitCode=0
Nov 02 14:57:17.826690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.826516    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"
Nov 02 14:57:17.826690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.826529    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" event={"ID":"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49","Type":"ContainerDied","Data":"60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57"}
Nov 02 14:57:17.826690 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.826576    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8" event={"ID":"59ea47e2-abbe-43d5-9225-3ce3b7c0dc49","Type":"ContainerDied","Data":"26a0b4c5226a74056b5dfb989b8be738222c520b43d97aa9dab7bce6fd154d14"}
Nov 02 14:57:17.828398 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.828371    5034 generic.go:356] "Generic (PLEG): container finished" podID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerID="ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb" exitCode=0
Nov 02 14:57:17.828526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.828421    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"
Nov 02 14:57:17.828526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.828452    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" event={"ID":"76c5fc52-0dad-47ac-ab36-f9ad186eabf8","Type":"ContainerDied","Data":"ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb"}
Nov 02 14:57:17.828526 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.828502    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr" event={"ID":"76c5fc52-0dad-47ac-ab36-f9ad186eabf8","Type":"ContainerDied","Data":"254fb95421836f78c39ed875dd2b768ba8848bba155863da62fc85e6c821e833"}
Nov 02 14:57:17.830233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.830213    5034 generic.go:356] "Generic (PLEG): container finished" podID="0bdca348-be51-47a6-bf14-32e7c181f160" containerID="3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166" exitCode=0
Nov 02 14:57:17.830345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.830272    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"
Nov 02 14:57:17.830345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.830296    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" event={"ID":"0bdca348-be51-47a6-bf14-32e7c181f160","Type":"ContainerDied","Data":"3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166"}
Nov 02 14:57:17.830345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.830340    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp" event={"ID":"0bdca348-be51-47a6-bf14-32e7c181f160","Type":"ContainerDied","Data":"3108ab13ef19075e9710ac25f064983723a2c06af1d773c7b54acf8f1aa01386"}
Nov 02 14:57:17.843923 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.843905    5034 scope.go:122] "RemoveContainer" containerID="8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6"
Nov 02 14:57:17.844194 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:17.844171    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6\": container with ID starting with 8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6 not found: ID does not exist" containerID="8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6"
Nov 02 14:57:17.844254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.844198    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6"} err="failed to get container status \"8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6\": rpc error: code = NotFound desc = could not find container \"8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6\": container with ID starting with 8fb7c0f4c6da15b7e358ea8c811a353b4f92b0f41bbe5784ae165d096626eed6 not found: ID does not exist"
Nov 02 14:57:17.844254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.844215    5034 scope.go:122] "RemoveContainer" containerID="60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57"
Nov 02 14:57:17.855578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.855200    5034 scope.go:122] "RemoveContainer" containerID="60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57"
Nov 02 14:57:17.855578 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:17.855457    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57\": container with ID starting with 60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57 not found: ID does not exist" containerID="60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57"
Nov 02 14:57:17.855578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.855479    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57"} err="failed to get container status \"60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57\": rpc error: code = NotFound desc = could not find container \"60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57\": container with ID starting with 60f52711caddba4ba2aa73fbca8164bbdf589271ca2b561e81620ca9c6ad8a57 not found: ID does not exist"
Nov 02 14:57:17.855578 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.855494    5034 scope.go:122] "RemoveContainer" containerID="ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb"
Nov 02 14:57:17.865382 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.865364    5034 scope.go:122] "RemoveContainer" containerID="ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb"
Nov 02 14:57:17.865634 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:17.865616    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb\": container with ID starting with ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb not found: ID does not exist" containerID="ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb"
Nov 02 14:57:17.865702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.865641    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb"} err="failed to get container status \"ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb\": rpc error: code = NotFound desc = could not find container \"ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb\": container with ID starting with ddbb0ff57d58c2ff8c8b9253bae6bf1b0d473683831e33867e6bbb68de1213cb not found: ID does not exist"
Nov 02 14:57:17.865702 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.865655    5034 scope.go:122] "RemoveContainer" containerID="3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166"
Nov 02 14:57:17.875991 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.875971    5034 scope.go:122] "RemoveContainer" containerID="3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166"
Nov 02 14:57:17.876283 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:17.876262    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166\": container with ID starting with 3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166 not found: ID does not exist" containerID="3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166"
Nov 02 14:57:17.876364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.876291    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166"} err="failed to get container status \"3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166\": rpc error: code = NotFound desc = could not find container \"3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166\": container with ID starting with 3c52543f695a3d3f92c9a156689ee382797f112061d6a3805b2771c37368e166 not found: ID does not exist"
Nov 02 14:57:17.902535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.902462    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"]
Nov 02 14:57:17.909249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.909216    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-n62n9"]
Nov 02 14:57:17.994589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:17.994560    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"]
Nov 02 14:57:18.000488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:18.000461    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-l2jwp"]
Nov 02 14:57:18.031871 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:18.031841    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"]
Nov 02 14:57:18.042712 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:18.042686    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-vgfz8"]
Nov 02 14:57:18.085677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:18.084216    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"]
Nov 02 14:57:18.088914 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:18.088884    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-k7bdr"]
Nov 02 14:57:18.414886 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:18.414844    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:57:19.396705 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:19.396674    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" path="/var/lib/kubelet/pods/0bdca348-be51-47a6-bf14-32e7c181f160/volumes"
Nov 02 14:57:19.397195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:19.397180    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" path="/var/lib/kubelet/pods/59ea47e2-abbe-43d5-9225-3ce3b7c0dc49/volumes"
Nov 02 14:57:19.397605 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:19.397592    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" path="/var/lib/kubelet/pods/76c5fc52-0dad-47ac-ab36-f9ad186eabf8/volumes"
Nov 02 14:57:19.398362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:19.398348    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" path="/var/lib/kubelet/pods/e1666e4c-a00a-408b-8fd5-4570510a96c4/volumes"
Nov 02 14:57:19.865156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:19.865111    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867" podUID="bce3a86b-ddf4-4b91-a2fb-909ecb45c052" containerName="payload-pre-processing" containerID="cri-o://2959f62e1ef0dd231f1d9531192da91ce88d356921fde1c7dfd61653955dacf7" gracePeriod=25
Nov 02 14:57:20.099541 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.099494    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv" podUID="c830edf9-caad-47fd-88aa-1c1e24e50790" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:20.803300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.803273    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:57:20.825063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.825040    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:57:20.863559 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.863532    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7597dc-d18f-46e0-b4df-6242defcbbe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7597dc-d18f-46e0-b4df-6242defcbbe6-plugins-config-volume\") pod \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\" (UID: \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\") "
Nov 02 14:57:20.863734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.863627    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7597dc-d18f-46e0-b4df-6242defcbbe6-kube-api-access-fhrlx\" (UniqueName: \"kubernetes.io/projected/fd7597dc-d18f-46e0-b4df-6242defcbbe6-kube-api-access-fhrlx\") pod \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\" (UID: \"fd7597dc-d18f-46e0-b4df-6242defcbbe6\") "
Nov 02 14:57:20.863734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.863674    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c830edf9-caad-47fd-88aa-1c1e24e50790-kube-api-access-s9j58\" (UniqueName: \"kubernetes.io/projected/c830edf9-caad-47fd-88aa-1c1e24e50790-kube-api-access-s9j58\") pod \"c830edf9-caad-47fd-88aa-1c1e24e50790\" (UID: \"c830edf9-caad-47fd-88aa-1c1e24e50790\") "
Nov 02 14:57:20.863734 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.863696    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c830edf9-caad-47fd-88aa-1c1e24e50790-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c830edf9-caad-47fd-88aa-1c1e24e50790-plugins-config-volume\") pod \"c830edf9-caad-47fd-88aa-1c1e24e50790\" (UID: \"c830edf9-caad-47fd-88aa-1c1e24e50790\") "
Nov 02 14:57:20.864345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.864000    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7597dc-d18f-46e0-b4df-6242defcbbe6-plugins-config-volume" pod "fd7597dc-d18f-46e0-b4df-6242defcbbe6" (UID: "fd7597dc-d18f-46e0-b4df-6242defcbbe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:20.864345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.864309    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd7597dc-d18f-46e0-b4df-6242defcbbe6-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:20.864703 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.864636    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c830edf9-caad-47fd-88aa-1c1e24e50790-plugins-config-volume" pod "c830edf9-caad-47fd-88aa-1c1e24e50790" (UID: "c830edf9-caad-47fd-88aa-1c1e24e50790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:20.866498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.866471    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7597dc-d18f-46e0-b4df-6242defcbbe6-kube-api-access-fhrlx" pod "fd7597dc-d18f-46e0-b4df-6242defcbbe6" (UID: "fd7597dc-d18f-46e0-b4df-6242defcbbe6"). InnerVolumeSpecName "kube-api-access-fhrlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:20.867053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.867017    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c830edf9-caad-47fd-88aa-1c1e24e50790-kube-api-access-s9j58" pod "c830edf9-caad-47fd-88aa-1c1e24e50790" (UID: "c830edf9-caad-47fd-88aa-1c1e24e50790"). InnerVolumeSpecName "kube-api-access-s9j58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:20.868836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.868813    5034 generic.go:356] "Generic (PLEG): container finished" podID="c830edf9-caad-47fd-88aa-1c1e24e50790" containerID="f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24" exitCode=0
Nov 02 14:57:20.868936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.868862    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"
Nov 02 14:57:20.868936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.868916    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv" event={"ID":"c830edf9-caad-47fd-88aa-1c1e24e50790","Type":"ContainerDied","Data":"f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24"}
Nov 02 14:57:20.869051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.868980    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv" event={"ID":"c830edf9-caad-47fd-88aa-1c1e24e50790","Type":"ContainerDied","Data":"6b6f2e586606894258f452a13c273aabb1f3c5c61610ffdbad00a5e16fea5690"}
Nov 02 14:57:20.869051 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.869012    5034 scope.go:122] "RemoveContainer" containerID="f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24"
Nov 02 14:57:20.870968 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.870942    5034 generic.go:356] "Generic (PLEG): container finished" podID="fd7597dc-d18f-46e0-b4df-6242defcbbe6" containerID="284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3" exitCode=0
Nov 02 14:57:20.871095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.870991    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"
Nov 02 14:57:20.871160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.871089    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp" event={"ID":"fd7597dc-d18f-46e0-b4df-6242defcbbe6","Type":"ContainerDied","Data":"284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3"}
Nov 02 14:57:20.871160 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.871133    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp" event={"ID":"fd7597dc-d18f-46e0-b4df-6242defcbbe6","Type":"ContainerDied","Data":"ecae0f2aef60126d58b3dc5f56ea302af7c80f1fa007a64fef5a5a0f929a7398"}
Nov 02 14:57:20.888297 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.888274    5034 scope.go:122] "RemoveContainer" containerID="f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24"
Nov 02 14:57:20.888588 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:20.888560    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24\": container with ID starting with f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24 not found: ID does not exist" containerID="f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24"
Nov 02 14:57:20.888656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.888600    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24"} err="failed to get container status \"f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24\": rpc error: code = NotFound desc = could not find container \"f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24\": container with ID starting with f16af891b1959a593b4229f181b1914223be82ef979904229e117270f7556d24 not found: ID does not exist"
Nov 02 14:57:20.888656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.888622    5034 scope.go:122] "RemoveContainer" containerID="284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3"
Nov 02 14:57:20.909983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.909953    5034 scope.go:122] "RemoveContainer" containerID="284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3"
Nov 02 14:57:20.910364 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:20.910336    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3\": container with ID starting with 284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3 not found: ID does not exist" containerID="284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3"
Nov 02 14:57:20.910470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.910374    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3"} err="failed to get container status \"284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3\": rpc error: code = NotFound desc = could not find container \"284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3\": container with ID starting with 284f57809ed7c5a1c2132be81d24b32241881350ef81767632b007b04c6308a3 not found: ID does not exist"
Nov 02 14:57:20.936284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.936254    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"]
Nov 02 14:57:20.944949 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.944926    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ee3eca-65fc8c5869-x7vgp"]
Nov 02 14:57:20.965287 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.965250    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhrlx\" (UniqueName: \"kubernetes.io/projected/fd7597dc-d18f-46e0-b4df-6242defcbbe6-kube-api-access-fhrlx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:20.965390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.965292    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9j58\" (UniqueName: \"kubernetes.io/projected/c830edf9-caad-47fd-88aa-1c1e24e50790-kube-api-access-s9j58\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:20.965390 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.965309    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c830edf9-caad-47fd-88aa-1c1e24e50790-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:20.981508 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.981485    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"]
Nov 02 14:57:20.984825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:20.984802    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ee3eca-7b76fcccd7-cmpwv"]
Nov 02 14:57:21.395394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:21.395361    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c830edf9-caad-47fd-88aa-1c1e24e50790" path="/var/lib/kubelet/pods/c830edf9-caad-47fd-88aa-1c1e24e50790/volumes"
Nov 02 14:57:21.395827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:21.395813    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7597dc-d18f-46e0-b4df-6242defcbbe6" path="/var/lib/kubelet/pods/fd7597dc-d18f-46e0-b4df-6242defcbbe6/volumes"
Nov 02 14:57:23.042663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:23.042632    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"]
Nov 02 14:57:23.076374 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:23.076346    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"]
Nov 02 14:57:24.742289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:24.742254    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867" podUID="bce3a86b-ddf4-4b91-a2fb-909ecb45c052" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:26.258185 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.258128    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:57:26.322561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.322534    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:57:26.380802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.378011    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac001bf-2ca9-4400-a244-f4404bf2351c-kube-api-access-4rlpq\" (UniqueName: \"kubernetes.io/projected/4ac001bf-2ca9-4400-a244-f4404bf2351c-kube-api-access-4rlpq\") pod \"4ac001bf-2ca9-4400-a244-f4404bf2351c\" (UID: \"4ac001bf-2ca9-4400-a244-f4404bf2351c\") "
Nov 02 14:57:26.380802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.378099    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac001bf-2ca9-4400-a244-f4404bf2351c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac001bf-2ca9-4400-a244-f4404bf2351c-plugins-config-volume\") pod \"4ac001bf-2ca9-4400-a244-f4404bf2351c\" (UID: \"4ac001bf-2ca9-4400-a244-f4404bf2351c\") "
Nov 02 14:57:26.384844 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.384139    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac001bf-2ca9-4400-a244-f4404bf2351c-plugins-config-volume" pod "4ac001bf-2ca9-4400-a244-f4404bf2351c" (UID: "4ac001bf-2ca9-4400-a244-f4404bf2351c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:26.387091 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.387063    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac001bf-2ca9-4400-a244-f4404bf2351c-kube-api-access-4rlpq" pod "4ac001bf-2ca9-4400-a244-f4404bf2351c" (UID: "4ac001bf-2ca9-4400-a244-f4404bf2351c"). InnerVolumeSpecName "kube-api-access-4rlpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:26.479581 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.479552    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6c78f1-4137-4146-958b-726d999c64b5-kube-api-access-7rv9g\" (UniqueName: \"kubernetes.io/projected/5b6c78f1-4137-4146-958b-726d999c64b5-kube-api-access-7rv9g\") pod \"5b6c78f1-4137-4146-958b-726d999c64b5\" (UID: \"5b6c78f1-4137-4146-958b-726d999c64b5\") "
Nov 02 14:57:26.479772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.479679    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6c78f1-4137-4146-958b-726d999c64b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6c78f1-4137-4146-958b-726d999c64b5-plugins-config-volume\") pod \"5b6c78f1-4137-4146-958b-726d999c64b5\" (UID: \"5b6c78f1-4137-4146-958b-726d999c64b5\") "
Nov 02 14:57:26.480107 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.480078    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6c78f1-4137-4146-958b-726d999c64b5-plugins-config-volume" pod "5b6c78f1-4137-4146-958b-726d999c64b5" (UID: "5b6c78f1-4137-4146-958b-726d999c64b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:26.480267 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.480247    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rlpq\" (UniqueName: \"kubernetes.io/projected/4ac001bf-2ca9-4400-a244-f4404bf2351c-kube-api-access-4rlpq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:26.480334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.480272    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac001bf-2ca9-4400-a244-f4404bf2351c-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:26.480334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.480290    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b6c78f1-4137-4146-958b-726d999c64b5-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:26.482066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.482030    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6c78f1-4137-4146-958b-726d999c64b5-kube-api-access-7rv9g" pod "5b6c78f1-4137-4146-958b-726d999c64b5" (UID: "5b6c78f1-4137-4146-958b-726d999c64b5"). InnerVolumeSpecName "kube-api-access-7rv9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:26.583608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.583516    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rv9g\" (UniqueName: \"kubernetes.io/projected/5b6c78f1-4137-4146-958b-726d999c64b5-kube-api-access-7rv9g\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:26.922901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.922815    5034 generic.go:356] "Generic (PLEG): container finished" podID="5b6c78f1-4137-4146-958b-726d999c64b5" containerID="96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b" exitCode=0
Nov 02 14:57:26.922901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.922863    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"
Nov 02 14:57:26.922901 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.922865    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs" event={"ID":"5b6c78f1-4137-4146-958b-726d999c64b5","Type":"ContainerDied","Data":"96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b"}
Nov 02 14:57:26.923233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.922945    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs" event={"ID":"5b6c78f1-4137-4146-958b-726d999c64b5","Type":"ContainerDied","Data":"18eaeb2cbd876a5d68169739eda46edc2be303c3f8bb95b468982f80b108ece7"}
Nov 02 14:57:26.923233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.922971    5034 scope.go:122] "RemoveContainer" containerID="96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b"
Nov 02 14:57:26.924566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.924537    5034 generic.go:356] "Generic (PLEG): container finished" podID="4ac001bf-2ca9-4400-a244-f4404bf2351c" containerID="c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c" exitCode=0
Nov 02 14:57:26.924686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.924599    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz" event={"ID":"4ac001bf-2ca9-4400-a244-f4404bf2351c","Type":"ContainerDied","Data":"c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c"}
Nov 02 14:57:26.924686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.924615    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"
Nov 02 14:57:26.924686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.924635    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz" event={"ID":"4ac001bf-2ca9-4400-a244-f4404bf2351c","Type":"ContainerDied","Data":"d4d06e5dc2fe7aa5064e546ed2071f813543df0f12d23fffa570df584d880a59"}
Nov 02 14:57:26.936324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.936305    5034 scope.go:122] "RemoveContainer" containerID="96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b"
Nov 02 14:57:26.936682 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:26.936661    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b\": container with ID starting with 96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b not found: ID does not exist" containerID="96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b"
Nov 02 14:57:26.936753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.936688    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b"} err="failed to get container status \"96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b\": rpc error: code = NotFound desc = could not find container \"96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b\": container with ID starting with 96bdc9e9df581142bdc8e6a73343297d73183c2f528b14edb7cf8b9d80d8a85b not found: ID does not exist"
Nov 02 14:57:26.936753 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.936704    5034 scope.go:122] "RemoveContainer" containerID="c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c"
Nov 02 14:57:26.946381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.946361    5034 scope.go:122] "RemoveContainer" containerID="c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c"
Nov 02 14:57:26.946662 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:26.946637    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c\": container with ID starting with c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c not found: ID does not exist" containerID="c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c"
Nov 02 14:57:26.946733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.946670    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c"} err="failed to get container status \"c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c\": rpc error: code = NotFound desc = could not find container \"c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c\": container with ID starting with c2c18460fefa19f2cb46c08d63ce6e107a4c66006349344bbc656e980de5804c not found: ID does not exist"
Nov 02 14:57:26.980500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.980474    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"]
Nov 02 14:57:26.988025 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:26.988000    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-5f98a6-64dd8c7b4f-9lzcs"]
Nov 02 14:57:27.034214 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:27.034191    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"]
Nov 02 14:57:27.037457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:27.037435    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-5f98a6-59f95cf747-gscpz"]
Nov 02 14:57:27.395848 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:27.395816    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac001bf-2ca9-4400-a244-f4404bf2351c" path="/var/lib/kubelet/pods/4ac001bf-2ca9-4400-a244-f4404bf2351c/volumes"
Nov 02 14:57:27.396252 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:27.396237    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6c78f1-4137-4146-958b-726d999c64b5" path="/var/lib/kubelet/pods/5b6c78f1-4137-4146-958b-726d999c64b5/volumes"
Nov 02 14:57:28.043000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:28.042956    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm" podUID="17dd575a-b600-4a15-87e2-d211b76d7a40" containerName="payload-processing" containerID="cri-o://2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35" gracePeriod=25
Nov 02 14:57:28.077615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:28.077581    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j" podUID="6e59015f-06e6-4f1b-90bb-58d210e1a0d5" containerName="payload-pre-processing" containerID="cri-o://fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1" gracePeriod=25
Nov 02 14:57:34.742603 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:34.742560    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867" podUID="bce3a86b-ddf4-4b91-a2fb-909ecb45c052" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:36.892471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:36.892429    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm" podUID="17dd575a-b600-4a15-87e2-d211b76d7a40" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:36.893010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:36.892437    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j" podUID="6e59015f-06e6-4f1b-90bb-58d210e1a0d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:37.769482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:37.769446    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"]
Nov 02 14:57:37.809521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:37.809485    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"]
Nov 02 14:57:38.900248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:38.900214    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"]
Nov 02 14:57:38.900801 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:38.900418    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy" containerID="cri-o://7be0d0d64e49f0e486a906badced715b1414f1f83d074d7eade1bdc41505c7fc" gracePeriod=30
Nov 02 14:57:39.662117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:39.662083    5034 scope.go:122] "RemoveContainer" containerID="587c480e73d1a59cb0fe36cefb2d304fadc9a7335733563fc03bea80ff23d950"
Nov 02 14:57:40.038576 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:40.038536    5034 generic.go:356] "Generic (PLEG): container finished" podID="bce3a86b-ddf4-4b91-a2fb-909ecb45c052" containerID="2959f62e1ef0dd231f1d9531192da91ce88d356921fde1c7dfd61653955dacf7" exitCode=0
Nov 02 14:57:40.038999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:40.038599    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867" event={"ID":"bce3a86b-ddf4-4b91-a2fb-909ecb45c052","Type":"ContainerDied","Data":"2959f62e1ef0dd231f1d9531192da91ce88d356921fde1c7dfd61653955dacf7"}
Nov 02 14:57:40.169195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:40.169166    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:57:40.253102 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:40.253023    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-plugins-config-volume\") pod \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\" (UID: \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\") "
Nov 02 14:57:40.253102 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:40.253068    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-kube-api-access-kvvl7\" (UniqueName: \"kubernetes.io/projected/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-kube-api-access-kvvl7\") pod \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\" (UID: \"bce3a86b-ddf4-4b91-a2fb-909ecb45c052\") "
Nov 02 14:57:40.253397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:40.253373    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-plugins-config-volume" pod "bce3a86b-ddf4-4b91-a2fb-909ecb45c052" (UID: "bce3a86b-ddf4-4b91-a2fb-909ecb45c052"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:40.253562 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:40.253541    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:40.255322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:40.255296    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-kube-api-access-kvvl7" pod "bce3a86b-ddf4-4b91-a2fb-909ecb45c052" (UID: "bce3a86b-ddf4-4b91-a2fb-909ecb45c052"). InnerVolumeSpecName "kube-api-access-kvvl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:40.355341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:40.355283    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvl7\" (UniqueName: \"kubernetes.io/projected/bce3a86b-ddf4-4b91-a2fb-909ecb45c052-kube-api-access-kvvl7\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:41.046223 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:41.046194    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"
Nov 02 14:57:41.046655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:41.046231    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867" event={"ID":"bce3a86b-ddf4-4b91-a2fb-909ecb45c052","Type":"ContainerDied","Data":"87e68b8bf622030b06a1738e44449bb6374468801d3d7c9b7d5e4d707c3b786b"}
Nov 02 14:57:41.046655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:41.046294    5034 scope.go:122] "RemoveContainer" containerID="2959f62e1ef0dd231f1d9531192da91ce88d356921fde1c7dfd61653955dacf7"
Nov 02 14:57:41.116106 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:41.116075    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"]
Nov 02 14:57:41.127821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:41.127795    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a02c35-665b58bd67-ck867"]
Nov 02 14:57:41.219187 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:41.219153    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"]
Nov 02 14:57:41.219422 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:41.219386    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy" containerID="cri-o://9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9" gracePeriod=30
Nov 02 14:57:41.396010 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:41.395923    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bce3a86b-ddf4-4b91-a2fb-909ecb45c052" path="/var/lib/kubelet/pods/bce3a86b-ddf4-4b91-a2fb-909ecb45c052/volumes"
Nov 02 14:57:42.769984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:42.769944    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc" podUID="f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" containerName="payload-processing" containerID="cri-o://5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22" gracePeriod=25
Nov 02 14:57:42.810314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:42.810270    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq" podUID="eaf25039-006c-447b-bbc7-a418d169eb39" containerName="payload-pre-processing" containerID="cri-o://7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81" gracePeriod=25
Nov 02 14:57:44.089279 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.089235    5034 generic.go:356] "Generic (PLEG): container finished" podID="83bd004e-2935-4b86-86ad-66d267777053" containerID="7be0d0d64e49f0e486a906badced715b1414f1f83d074d7eade1bdc41505c7fc" exitCode=0
Nov 02 14:57:44.089815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.089500    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" event={"ID":"83bd004e-2935-4b86-86ad-66d267777053","Type":"ContainerDied","Data":"7be0d0d64e49f0e486a906badced715b1414f1f83d074d7eade1bdc41505c7fc"}
Nov 02 14:57:44.089815 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.089559    5034 scope.go:122] "RemoveContainer" containerID="81992a96fa8c1d7d59d976f6a107641bcea05e312a2ee54eed75833e62922330"
Nov 02 14:57:44.228995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.228969    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:57:44.310292 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310259    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-envoy\") pod \"83bd004e-2935-4b86-86ad-66d267777053\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") "
Nov 02 14:57:44.310479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310306    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-socket\") pod \"83bd004e-2935-4b86-86ad-66d267777053\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") "
Nov 02 14:57:44.310479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310397    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83bd004e-2935-4b86-86ad-66d267777053-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bd004e-2935-4b86-86ad-66d267777053-istiod-ca-cert\") pod \"83bd004e-2935-4b86-86ad-66d267777053\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") "
Nov 02 14:57:44.310479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310458    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-kube-api-access-m68sl\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-kube-api-access-m68sl\") pod \"83bd004e-2935-4b86-86ad-66d267777053\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") "
Nov 02 14:57:44.310662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310512    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-istio-token\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-istio-token\") pod \"83bd004e-2935-4b86-86ad-66d267777053\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") "
Nov 02 14:57:44.310662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310559    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-data\") pod \"83bd004e-2935-4b86-86ad-66d267777053\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") "
Nov 02 14:57:44.310662 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310631    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-credential-socket\") pod \"83bd004e-2935-4b86-86ad-66d267777053\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") "
Nov 02 14:57:44.310859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310682    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-certs\") pod \"83bd004e-2935-4b86-86ad-66d267777053\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") "
Nov 02 14:57:44.310859 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310747    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83bd004e-2935-4b86-86ad-66d267777053-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bd004e-2935-4b86-86ad-66d267777053-istio-podinfo\") pod \"83bd004e-2935-4b86-86ad-66d267777053\" (UID: \"83bd004e-2935-4b86-86ad-66d267777053\") "
Nov 02 14:57:44.310978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.310859    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bd004e-2935-4b86-86ad-66d267777053-istiod-ca-cert" pod "83bd004e-2935-4b86-86ad-66d267777053" (UID: "83bd004e-2935-4b86-86ad-66d267777053"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:44.311186 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.311163    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-certs" pod "83bd004e-2935-4b86-86ad-66d267777053" (UID: "83bd004e-2935-4b86-86ad-66d267777053"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:44.311481 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.311459    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bd004e-2935-4b86-86ad-66d267777053-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.311545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.311489    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.311545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.311497    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-credential-socket" pod "83bd004e-2935-4b86-86ad-66d267777053" (UID: "83bd004e-2935-4b86-86ad-66d267777053"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:44.311659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.311563    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-data" pod "83bd004e-2935-4b86-86ad-66d267777053" (UID: "83bd004e-2935-4b86-86ad-66d267777053"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:44.311659 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.311617    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-socket" pod "83bd004e-2935-4b86-86ad-66d267777053" (UID: "83bd004e-2935-4b86-86ad-66d267777053"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:44.312913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.312884    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-envoy" pod "83bd004e-2935-4b86-86ad-66d267777053" (UID: "83bd004e-2935-4b86-86ad-66d267777053"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:44.313839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.313762    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83bd004e-2935-4b86-86ad-66d267777053-istio-podinfo" pod "83bd004e-2935-4b86-86ad-66d267777053" (UID: "83bd004e-2935-4b86-86ad-66d267777053"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:44.314063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.314038    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-kube-api-access-m68sl" pod "83bd004e-2935-4b86-86ad-66d267777053" (UID: "83bd004e-2935-4b86-86ad-66d267777053"). InnerVolumeSpecName "kube-api-access-m68sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:44.314137 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.314064    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-istio-token" pod "83bd004e-2935-4b86-86ad-66d267777053" (UID: "83bd004e-2935-4b86-86ad-66d267777053"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:44.412763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.412723    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.412763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.412752    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bd004e-2935-4b86-86ad-66d267777053-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.412763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.412762    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.412763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.412770    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.412763 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.412802    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m68sl\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-kube-api-access-m68sl\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.413225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.412812    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83bd004e-2935-4b86-86ad-66d267777053-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.413225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.412820    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bd004e-2935-4b86-86ad-66d267777053-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:44.966217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:44.966182    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"]
Nov 02 14:57:45.097608 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:45.097580    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"
Nov 02 14:57:45.098044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:45.097616    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v" event={"ID":"83bd004e-2935-4b86-86ad-66d267777053","Type":"ContainerDied","Data":"4501e3c8cf06e00be43209315bdccb875cda895f512437dec5d535b3f92dc873"}
Nov 02 14:57:45.098044 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:45.097674    5034 scope.go:122] "RemoveContainer" containerID="7be0d0d64e49f0e486a906badced715b1414f1f83d074d7eade1bdc41505c7fc"
Nov 02 14:57:45.176140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:45.176100    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"]
Nov 02 14:57:45.190015 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:45.189985    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-21baa7-openshift-default-f7457b5bf-vqm9v"]
Nov 02 14:57:45.397396 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:45.397362    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83bd004e-2935-4b86-86ad-66d267777053" path="/var/lib/kubelet/pods/83bd004e-2935-4b86-86ad-66d267777053/volumes"
Nov 02 14:57:46.495305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.495246    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:57:46.643065 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643027    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-podinfo\") pod \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") "
Nov 02 14:57:46.643242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643166    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-certs\") pod \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") "
Nov 02 14:57:46.643242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643202    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-envoy\") pod \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") "
Nov 02 14:57:46.643348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643296    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istiod-ca-cert\") pod \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") "
Nov 02 14:57:46.643412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643342    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-token\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-token\") pod \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") "
Nov 02 14:57:46.643412 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643372    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-socket\") pod \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") "
Nov 02 14:57:46.643535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643422    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-data\") pod \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") "
Nov 02 14:57:46.643535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643435    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-certs" pod "81deadf7-ae3c-4c10-a7e1-40906f764a9f" (UID: "81deadf7-ae3c-4c10-a7e1-40906f764a9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:46.643535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643453    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-kube-api-access-8qvdr\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-kube-api-access-8qvdr\") pod \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") "
Nov 02 14:57:46.643535 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643499    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-credential-socket\") pod \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\" (UID: \"81deadf7-ae3c-4c10-a7e1-40906f764a9f\") "
Nov 02 14:57:46.643769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643667    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-socket" pod "81deadf7-ae3c-4c10-a7e1-40906f764a9f" (UID: "81deadf7-ae3c-4c10-a7e1-40906f764a9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:46.643769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643708    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-data" pod "81deadf7-ae3c-4c10-a7e1-40906f764a9f" (UID: "81deadf7-ae3c-4c10-a7e1-40906f764a9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:46.643769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643736    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istiod-ca-cert" pod "81deadf7-ae3c-4c10-a7e1-40906f764a9f" (UID: "81deadf7-ae3c-4c10-a7e1-40906f764a9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:46.644001 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.643893    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-credential-socket" pod "81deadf7-ae3c-4c10-a7e1-40906f764a9f" (UID: "81deadf7-ae3c-4c10-a7e1-40906f764a9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:46.644111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.644089    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:46.644183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.644112    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:46.644183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.644120    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:46.644183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.644128    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:46.644183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.644136    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:46.645533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.645501    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-podinfo" pod "81deadf7-ae3c-4c10-a7e1-40906f764a9f" (UID: "81deadf7-ae3c-4c10-a7e1-40906f764a9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:46.645698 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.645539    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-envoy" pod "81deadf7-ae3c-4c10-a7e1-40906f764a9f" (UID: "81deadf7-ae3c-4c10-a7e1-40906f764a9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:46.645978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.645955    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-kube-api-access-8qvdr" pod "81deadf7-ae3c-4c10-a7e1-40906f764a9f" (UID: "81deadf7-ae3c-4c10-a7e1-40906f764a9f"). InnerVolumeSpecName "kube-api-access-8qvdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:46.646072 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.645991    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-token" pod "81deadf7-ae3c-4c10-a7e1-40906f764a9f" (UID: "81deadf7-ae3c-4c10-a7e1-40906f764a9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:46.745593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.745534    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:46.745593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.745559    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qvdr\" (UniqueName: \"kubernetes.io/projected/81deadf7-ae3c-4c10-a7e1-40906f764a9f-kube-api-access-8qvdr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:46.745593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.745569    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:46.745593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.745577    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81deadf7-ae3c-4c10-a7e1-40906f764a9f-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:46.892449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.892413    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm" podUID="17dd575a-b600-4a15-87e2-d211b76d7a40" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:46.892449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:46.892437    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j" podUID="6e59015f-06e6-4f1b-90bb-58d210e1a0d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:47.081428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.081330    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq" podUID="eaf25039-006c-447b-bbc7-a418d169eb39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:47.117323 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.117279    5034 generic.go:356] "Generic (PLEG): container finished" podID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerID="9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9" exitCode=0
Nov 02 14:57:47.117515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.117338    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"
Nov 02 14:57:47.117515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.117359    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" event={"ID":"81deadf7-ae3c-4c10-a7e1-40906f764a9f","Type":"ContainerDied","Data":"9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9"}
Nov 02 14:57:47.117515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.117440    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf" event={"ID":"81deadf7-ae3c-4c10-a7e1-40906f764a9f","Type":"ContainerDied","Data":"33cac51c769d40114be3f76ec2b9b37a1f6cbad3341775159906f75a46af2d38"}
Nov 02 14:57:47.117515 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.117464    5034 scope.go:122] "RemoveContainer" containerID="9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9"
Nov 02 14:57:47.128492 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.128469    5034 scope.go:122] "RemoveContainer" containerID="9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9"
Nov 02 14:57:47.128766 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:47.128748    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9\": container with ID starting with 9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9 not found: ID does not exist" containerID="9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9"
Nov 02 14:57:47.128839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.128773    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9"} err="failed to get container status \"9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9\": rpc error: code = NotFound desc = could not find container \"9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9\": container with ID starting with 9eb8bdc68367298cf4e44912fe7067bbd0ae462573fed02f1149e135823c6fc9 not found: ID does not exist"
Nov 02 14:57:47.177551 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.177528    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"]
Nov 02 14:57:47.193132 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.193104    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-2cddf2-openshift-default-65c55b8dbf-l8qpf"]
Nov 02 14:57:47.395990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:47.395920    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" path="/var/lib/kubelet/pods/81deadf7-ae3c-4c10-a7e1-40906f764a9f/volumes"
Nov 02 14:57:48.121862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.121825    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc" podUID="f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:48.360184 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.360157    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:57:48.382295 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.382270    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:57:48.416081 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:48.416049    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:57:48.466273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.466243    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-kube-api-access-xb4vd\" (UniqueName: \"kubernetes.io/projected/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-kube-api-access-xb4vd\") pod \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\" (UID: \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\") "
Nov 02 14:57:48.466445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.466404    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-plugins-config-volume\") pod \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\" (UID: \"6e59015f-06e6-4f1b-90bb-58d210e1a0d5\") "
Nov 02 14:57:48.466445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.466432    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17dd575a-b600-4a15-87e2-d211b76d7a40-kube-api-access-pwkzj\" (UniqueName: \"kubernetes.io/projected/17dd575a-b600-4a15-87e2-d211b76d7a40-kube-api-access-pwkzj\") pod \"17dd575a-b600-4a15-87e2-d211b76d7a40\" (UID: \"17dd575a-b600-4a15-87e2-d211b76d7a40\") "
Nov 02 14:57:48.466591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.466462    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17dd575a-b600-4a15-87e2-d211b76d7a40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17dd575a-b600-4a15-87e2-d211b76d7a40-plugins-config-volume\") pod \"17dd575a-b600-4a15-87e2-d211b76d7a40\" (UID: \"17dd575a-b600-4a15-87e2-d211b76d7a40\") "
Nov 02 14:57:48.466803 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.466757    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-plugins-config-volume" pod "6e59015f-06e6-4f1b-90bb-58d210e1a0d5" (UID: "6e59015f-06e6-4f1b-90bb-58d210e1a0d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:48.466948 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.466924    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:48.467059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.466969    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17dd575a-b600-4a15-87e2-d211b76d7a40-plugins-config-volume" pod "17dd575a-b600-4a15-87e2-d211b76d7a40" (UID: "17dd575a-b600-4a15-87e2-d211b76d7a40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:48.468672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.468649    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17dd575a-b600-4a15-87e2-d211b76d7a40-kube-api-access-pwkzj" pod "17dd575a-b600-4a15-87e2-d211b76d7a40" (UID: "17dd575a-b600-4a15-87e2-d211b76d7a40"). InnerVolumeSpecName "kube-api-access-pwkzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:48.468872 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.468855    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-kube-api-access-xb4vd" pod "6e59015f-06e6-4f1b-90bb-58d210e1a0d5" (UID: "6e59015f-06e6-4f1b-90bb-58d210e1a0d5"). InnerVolumeSpecName "kube-api-access-xb4vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:48.568524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.568496    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwkzj\" (UniqueName: \"kubernetes.io/projected/17dd575a-b600-4a15-87e2-d211b76d7a40-kube-api-access-pwkzj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:48.568524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.568523    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17dd575a-b600-4a15-87e2-d211b76d7a40-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:48.568714 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:48.568534    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb4vd\" (UniqueName: \"kubernetes.io/projected/6e59015f-06e6-4f1b-90bb-58d210e1a0d5-kube-api-access-xb4vd\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:49.133255 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.133220    5034 generic.go:356] "Generic (PLEG): container finished" podID="17dd575a-b600-4a15-87e2-d211b76d7a40" containerID="2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35" exitCode=0
Nov 02 14:57:49.133710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.133284    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"
Nov 02 14:57:49.133710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.133306    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm" event={"ID":"17dd575a-b600-4a15-87e2-d211b76d7a40","Type":"ContainerDied","Data":"2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35"}
Nov 02 14:57:49.133710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.133367    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm" event={"ID":"17dd575a-b600-4a15-87e2-d211b76d7a40","Type":"ContainerDied","Data":"18e26ffc1f315207d00cd1ca83206fa4ae0cfdb2500e9426ef57adbcc5e5aa66"}
Nov 02 14:57:49.133710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.133396    5034 scope.go:122] "RemoveContainer" containerID="2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35"
Nov 02 14:57:49.135196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.135174    5034 generic.go:356] "Generic (PLEG): container finished" podID="6e59015f-06e6-4f1b-90bb-58d210e1a0d5" containerID="fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1" exitCode=0
Nov 02 14:57:49.135376 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.135215    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"
Nov 02 14:57:49.135468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.135231    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j" event={"ID":"6e59015f-06e6-4f1b-90bb-58d210e1a0d5","Type":"ContainerDied","Data":"fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1"}
Nov 02 14:57:49.135527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.135500    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j" event={"ID":"6e59015f-06e6-4f1b-90bb-58d210e1a0d5","Type":"ContainerDied","Data":"d909c230ec32261eaef52e073889c878210a60fc4f30187c7f972a362791aad4"}
Nov 02 14:57:49.147945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.147923    5034 scope.go:122] "RemoveContainer" containerID="2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35"
Nov 02 14:57:49.148254 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:49.148232    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35\": container with ID starting with 2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35 not found: ID does not exist" containerID="2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35"
Nov 02 14:57:49.148315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.148263    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35"} err="failed to get container status \"2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35\": rpc error: code = NotFound desc = could not find container \"2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35\": container with ID starting with 2e1f8cee5f364558d4f9f766237d4242456e2bf88dbd83cd2ed7d660d5702c35 not found: ID does not exist"
Nov 02 14:57:49.148315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.148284    5034 scope.go:122] "RemoveContainer" containerID="fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1"
Nov 02 14:57:49.158501 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.158481    5034 scope.go:122] "RemoveContainer" containerID="fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1"
Nov 02 14:57:49.158750 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:49.158733    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1\": container with ID starting with fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1 not found: ID does not exist" containerID="fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1"
Nov 02 14:57:49.158824 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.158756    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1"} err="failed to get container status \"fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1\": rpc error: code = NotFound desc = could not find container \"fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1\": container with ID starting with fd8301faae0811d27869bdd21a3f0f70d64ae918f2ca76fcc2de6d224686cef1 not found: ID does not exist"
Nov 02 14:57:49.208523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.208500    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"]
Nov 02 14:57:49.221898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.221878    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-21baa7-7f48f846d7-rcbtm"]
Nov 02 14:57:49.246285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.246260    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"]
Nov 02 14:57:49.251354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.251331    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-21baa7-86c8d9fbc-47v2j"]
Nov 02 14:57:49.394401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.394335    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17dd575a-b600-4a15-87e2-d211b76d7a40" path="/var/lib/kubelet/pods/17dd575a-b600-4a15-87e2-d211b76d7a40/volumes"
Nov 02 14:57:49.394790 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.394766    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e59015f-06e6-4f1b-90bb-58d210e1a0d5" path="/var/lib/kubelet/pods/6e59015f-06e6-4f1b-90bb-58d210e1a0d5/volumes"
Nov 02 14:57:49.966520 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:49.966476    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4" podUID="f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" containerName="payload-processing" containerID="cri-o://cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242" gracePeriod=25
Nov 02 14:57:52.251864 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:52.251832    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"]
Nov 02 14:57:52.252304 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:52.252065    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy" containerID="cri-o://38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973" gracePeriod=30
Nov 02 14:57:55.598554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:55.598237    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"]
Nov 02 14:57:55.598554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:55.598492    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy" containerID="cri-o://78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734" gracePeriod=30
Nov 02 14:57:56.608203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.608167    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"]
Nov 02 14:57:56.609030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609000    5034 cpu_manager.go:414] "containerMap: removing container" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy"
Nov 02 14:57:56.609030 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609029    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609047    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609054    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609074    5034 cpu_manager.go:414] "containerMap: removing container" podUID="30e257b7-094d-48da-a548-3ec863b5f148" containerName="payload-pre-processing"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609083    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e257b7-094d-48da-a548-3ec863b5f148" containerName="payload-pre-processing"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609101    5034 cpu_manager.go:414] "containerMap: removing container" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609108    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609136    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609144    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609159    5034 cpu_manager.go:414] "containerMap: removing container" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609168    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609183    5034 cpu_manager.go:414] "containerMap: removing container" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609191    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609209    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bce3a86b-ddf4-4b91-a2fb-909ecb45c052" containerName="payload-pre-processing"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609217    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bce3a86b-ddf4-4b91-a2fb-909ecb45c052" containerName="payload-pre-processing"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609233    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609240    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609250    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4ac001bf-2ca9-4400-a244-f4404bf2351c" containerName="payload-pre-processing"
Nov 02 14:57:56.609261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609257    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac001bf-2ca9-4400-a244-f4404bf2351c" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609274    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6e59015f-06e6-4f1b-90bb-58d210e1a0d5" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609282    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e59015f-06e6-4f1b-90bb-58d210e1a0d5" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609295    5034 cpu_manager.go:414] "containerMap: removing container" podUID="17dd575a-b600-4a15-87e2-d211b76d7a40" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609303    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17dd575a-b600-4a15-87e2-d211b76d7a40" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609320    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c0d60b57-6e94-4c11-9128-f9668c124b9f" containerName="revoke-keys"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609327    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d60b57-6e94-4c11-9128-f9668c124b9f" containerName="revoke-keys"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609338    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609345    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609362    5034 cpu_manager.go:414] "containerMap: removing container" podUID="0604c248-3e5a-4de8-8772-932e884a4326" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609368    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0604c248-3e5a-4de8-8772-932e884a4326" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609380    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c830edf9-caad-47fd-88aa-1c1e24e50790" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609388    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c830edf9-caad-47fd-88aa-1c1e24e50790" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609402    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609411    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609425    5034 cpu_manager.go:414] "containerMap: removing container" podUID="fd7597dc-d18f-46e0-b4df-6242defcbbe6" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609431    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7597dc-d18f-46e0-b4df-6242defcbbe6" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609440    5034 cpu_manager.go:414] "containerMap: removing container" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609446    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609457    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5b6c78f1-4137-4146-958b-726d999c64b5" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609464    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6c78f1-4137-4146-958b-726d999c64b5" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609676    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ea47e2-abbe-43d5-9225-3ce3b7c0dc49" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609702    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609712    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="81deadf7-ae3c-4c10-a7e1-40906f764a9f" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609722    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609733    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609741    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bce3a86b-ddf4-4b91-a2fb-909ecb45c052" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609749    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac001bf-2ca9-4400-a244-f4404bf2351c" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609762    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e59015f-06e6-4f1b-90bb-58d210e1a0d5" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609771    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="17dd575a-b600-4a15-87e2-d211b76d7a40" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609800    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609809    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d60b57-6e94-4c11-9128-f9668c124b9f" containerName="revoke-keys"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609820    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3805ee4d-ff1c-47f8-bd49-00b7b21e4eac" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609832    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0604c248-3e5a-4de8-8772-932e884a4326" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609840    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c830edf9-caad-47fd-88aa-1c1e24e50790" containerName="payload-pre-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609848    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1666e4c-a00a-408b-8fd5-4570510a96c4" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609859    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7597dc-d18f-46e0-b4df-6242defcbbe6" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609869    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="76c5fc52-0dad-47ac-ab36-f9ad186eabf8" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609877    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6c78f1-4137-4146-958b-726d999c64b5" containerName="payload-processing"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609885    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609895    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdca348-be51-47a6-bf14-32e7c181f160" containerName="istio-proxy"
Nov 02 14:57:56.610195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.609907    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e257b7-094d-48da-a548-3ec863b5f148" containerName="payload-pre-processing"
Nov 02 14:57:56.626196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.626165    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.635420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.635389    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"]
Nov 02 14:57:56.643772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.643749    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:57:56.644040 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.644021    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:57:56.657609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.657591    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:57:56.657609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.657600    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:57:56.775500 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.775471    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hvd\" (UniqueName: \"kubernetes.io/projected/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-kube-api-access-f4hvd\") pod \"maas-api-revoke-sub-3a3e2934-6w6dm\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.775678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.775518    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3e2934-6w6dm\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.775678 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.775545    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3e2934-6w6dm\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.877389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.877316    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hvd\" (UniqueName: \"kubernetes.io/projected/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-kube-api-access-f4hvd\") pod \"maas-api-revoke-sub-3a3e2934-6w6dm\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.877389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.877374    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3e2934-6w6dm\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.877582 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.877406    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3e2934-6w6dm\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.878248 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.878222    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3e2934-6w6dm\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.879906 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.879883    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3e2934-6w6dm\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.902352 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.902329    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hvd\" (UniqueName: \"kubernetes.io/projected/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-kube-api-access-f4hvd\") pod \"maas-api-revoke-sub-3a3e2934-6w6dm\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:56.944003 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:56.943982    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:57:57.089403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.088901    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq" podUID="eaf25039-006c-447b-bbc7-a418d169eb39" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:57.278923 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:57:57.278894    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136896ee_fa2e_4c92_9f6e_f2b18f3ce509.slice/crio-03255ffea6698f7104b2fcc8e1bdbf5a17d59bbb32ae8cd3451947d105d489a9 WatchSource:0}: Error finding container 03255ffea6698f7104b2fcc8e1bdbf5a17d59bbb32ae8cd3451947d105d489a9: Status 404 returned error can't find the container with id 03255ffea6698f7104b2fcc8e1bdbf5a17d59bbb32ae8cd3451947d105d489a9
Nov 02 14:57:57.280327 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.280302    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"]
Nov 02 14:57:57.354167 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.354134    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4" podUID="f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:57.521488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.521420    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:57:57.593746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.593670    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-podinfo\") pod \"5cab00b6-3287-4f34-a7e3-23fc811d615c\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") "
Nov 02 14:57:57.593930 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.593896    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-data\") pod \"5cab00b6-3287-4f34-a7e3-23fc811d615c\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") "
Nov 02 14:57:57.594000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.593952    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-credential-socket\") pod \"5cab00b6-3287-4f34-a7e3-23fc811d615c\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") "
Nov 02 14:57:57.594000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.593990    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-envoy\") pod \"5cab00b6-3287-4f34-a7e3-23fc811d615c\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") "
Nov 02 14:57:57.594127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.594031    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-kube-api-access-jz4p5\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-kube-api-access-jz4p5\") pod \"5cab00b6-3287-4f34-a7e3-23fc811d615c\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") "
Nov 02 14:57:57.594127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.594080    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-token\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-token\") pod \"5cab00b6-3287-4f34-a7e3-23fc811d615c\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") "
Nov 02 14:57:57.594247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.594131    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cab00b6-3287-4f34-a7e3-23fc811d615c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cab00b6-3287-4f34-a7e3-23fc811d615c-istiod-ca-cert\") pod \"5cab00b6-3287-4f34-a7e3-23fc811d615c\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") "
Nov 02 14:57:57.594372 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.594351    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-socket\") pod \"5cab00b6-3287-4f34-a7e3-23fc811d615c\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") "
Nov 02 14:57:57.594435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.594413    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-certs\") pod \"5cab00b6-3287-4f34-a7e3-23fc811d615c\" (UID: \"5cab00b6-3287-4f34-a7e3-23fc811d615c\") "
Nov 02 14:57:57.597016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.596597    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cab00b6-3287-4f34-a7e3-23fc811d615c-istiod-ca-cert" pod "5cab00b6-3287-4f34-a7e3-23fc811d615c" (UID: "5cab00b6-3287-4f34-a7e3-23fc811d615c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:57:57.597016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.596926    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-socket" pod "5cab00b6-3287-4f34-a7e3-23fc811d615c" (UID: "5cab00b6-3287-4f34-a7e3-23fc811d615c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.597016 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.596986    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-credential-socket" pod "5cab00b6-3287-4f34-a7e3-23fc811d615c" (UID: "5cab00b6-3287-4f34-a7e3-23fc811d615c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.597609 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.597586    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-data" pod "5cab00b6-3287-4f34-a7e3-23fc811d615c" (UID: "5cab00b6-3287-4f34-a7e3-23fc811d615c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.597817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.597796    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-certs" pod "5cab00b6-3287-4f34-a7e3-23fc811d615c" (UID: "5cab00b6-3287-4f34-a7e3-23fc811d615c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.599242 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.599218    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-envoy" pod "5cab00b6-3287-4f34-a7e3-23fc811d615c" (UID: "5cab00b6-3287-4f34-a7e3-23fc811d615c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:57:57.599353 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.599254    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-token" pod "5cab00b6-3287-4f34-a7e3-23fc811d615c" (UID: "5cab00b6-3287-4f34-a7e3-23fc811d615c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:57.600700 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.600679    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-podinfo" pod "5cab00b6-3287-4f34-a7e3-23fc811d615c" (UID: "5cab00b6-3287-4f34-a7e3-23fc811d615c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:57:57.601314 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.601273    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-kube-api-access-jz4p5" pod "5cab00b6-3287-4f34-a7e3-23fc811d615c" (UID: "5cab00b6-3287-4f34-a7e3-23fc811d615c"). InnerVolumeSpecName "kube-api-access-jz4p5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:57:57.697131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.696383    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.697131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.696412    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.697131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.696421    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.697131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.696429    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.697131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.696438    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.697131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.696447    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.697131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.696457    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4p5\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-kube-api-access-jz4p5\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.697131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.696465    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cab00b6-3287-4f34-a7e3-23fc811d615c-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:57.697131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:57.696473    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cab00b6-3287-4f34-a7e3-23fc811d615c-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:57:58.122839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.122733    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc" podUID="f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:57:58.203225 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.203192    5034 generic.go:356] "Generic (PLEG): container finished" podID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerID="38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973" exitCode=0
Nov 02 14:57:58.203378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.203231    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" event={"ID":"5cab00b6-3287-4f34-a7e3-23fc811d615c","Type":"ContainerDied","Data":"38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973"}
Nov 02 14:57:58.203378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.203287    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s" event={"ID":"5cab00b6-3287-4f34-a7e3-23fc811d615c","Type":"ContainerDied","Data":"8e2b190321b307dfb16c201b079bd5d3c69acf3fbf05765ba2f05d9de12bd65d"}
Nov 02 14:57:58.203378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.203310    5034 scope.go:122] "RemoveContainer" containerID="38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973"
Nov 02 14:57:58.203378 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.203248    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"
Nov 02 14:57:58.205397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.205369    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm" event={"ID":"136896ee-fa2e-4c92-9f6e-f2b18f3ce509","Type":"ContainerStarted","Data":"3b754a05e5e703a1c83be296c7278db869564c07cde07756758400836015d3e2"}
Nov 02 14:57:58.205518 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.205422    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm" event={"ID":"136896ee-fa2e-4c92-9f6e-f2b18f3ce509","Type":"ContainerStarted","Data":"03255ffea6698f7104b2fcc8e1bdbf5a17d59bbb32ae8cd3451947d105d489a9"}
Nov 02 14:57:58.214047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.214020    5034 scope.go:122] "RemoveContainer" containerID="38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973"
Nov 02 14:57:58.214309 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:57:58.214289    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973\": container with ID starting with 38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973 not found: ID does not exist" containerID="38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973"
Nov 02 14:57:58.214389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.214317    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973"} err="failed to get container status \"38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973\": rpc error: code = NotFound desc = could not find container \"38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973\": container with ID starting with 38e16d4de5bd823d7c1a620528348d1e8b459ccf065917b74aa0c92f4ca05973 not found: ID does not exist"
Nov 02 14:57:58.263975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.263944    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"]
Nov 02 14:57:58.277855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.277825    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-960a68-openshift-default-787d7477dc-p4q5s"]
Nov 02 14:57:58.345180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:58.345125    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm" podStartSLOduration=2.34510787 podStartE2EDuration="2.34510787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:58.339500988 +0000 UTC m=+4469.090805426" watchObservedRunningTime="2026-11-02 14:57:58.34510787 +0000 UTC m=+4469.096412309"
Nov 02 14:57:59.214950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:59.214910    5034 generic.go:356] "Generic (PLEG): container finished" podID="136896ee-fa2e-4c92-9f6e-f2b18f3ce509" containerID="3b754a05e5e703a1c83be296c7278db869564c07cde07756758400836015d3e2" exitCode=0
Nov 02 14:57:59.215423 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:59.214970    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm" event={"ID":"136896ee-fa2e-4c92-9f6e-f2b18f3ce509","Type":"ContainerDied","Data":"3b754a05e5e703a1c83be296c7278db869564c07cde07756758400836015d3e2"}
Nov 02 14:57:59.396591 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:59.396555    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" path="/var/lib/kubelet/pods/5cab00b6-3287-4f34-a7e3-23fc811d615c/volumes"
Nov 02 14:57:59.491730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:59.491700    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:57:59.491936 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:57:59.491751    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:58:00.403231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.403196    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:58:00.430172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.430144    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-openshift-ca\") pod \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") "
Nov 02 14:58:00.430334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.430259    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-xks-ca\") pod \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") "
Nov 02 14:58:00.430406 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.430348    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-kube-api-access-f4hvd\" (UniqueName: \"kubernetes.io/projected/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-kube-api-access-f4hvd\") pod \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\" (UID: \"136896ee-fa2e-4c92-9f6e-f2b18f3ce509\") "
Nov 02 14:58:00.430542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.430514    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-openshift-ca" pod "136896ee-fa2e-4c92-9f6e-f2b18f3ce509" (UID: "136896ee-fa2e-4c92-9f6e-f2b18f3ce509"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:00.430836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.430816    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:00.432525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.432483    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-xks-ca" pod "136896ee-fa2e-4c92-9f6e-f2b18f3ce509" (UID: "136896ee-fa2e-4c92-9f6e-f2b18f3ce509"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:00.432615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.432531    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-kube-api-access-f4hvd" pod "136896ee-fa2e-4c92-9f6e-f2b18f3ce509" (UID: "136896ee-fa2e-4c92-9f6e-f2b18f3ce509"). InnerVolumeSpecName "kube-api-access-f4hvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:00.531756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.531720    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:00.531756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.531757    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4hvd\" (UniqueName: \"kubernetes.io/projected/136896ee-fa2e-4c92-9f6e-f2b18f3ce509-kube-api-access-f4hvd\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:00.870808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.870765    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:58:00.936772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.936742    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-certs\") pod \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") "
Nov 02 14:58:00.936772 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.936801    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-envoy\") pod \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") "
Nov 02 14:58:00.937048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.936839    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-credential-socket\") pod \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") "
Nov 02 14:58:00.937048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.936908    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-podinfo\") pod \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") "
Nov 02 14:58:00.937048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.936968    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-socket\") pod \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") "
Nov 02 14:58:00.937048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937012    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-kube-api-access-zc6s9\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-kube-api-access-zc6s9\") pod \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") "
Nov 02 14:58:00.937284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937054    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-certs" pod "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" (UID: "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:00.937284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937062    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istiod-ca-cert\") pod \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") "
Nov 02 14:58:00.937284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937130    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-token\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-token\") pod \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") "
Nov 02 14:58:00.937284 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937171    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-data\") pod \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\" (UID: \"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc\") "
Nov 02 14:58:00.937440 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937386    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istiod-ca-cert" pod "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" (UID: "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:00.937594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937566    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-socket" pod "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" (UID: "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:00.937594 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937577    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-credential-socket" pod "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" (UID: "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:00.937738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937718    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:00.937738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937719    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-data" pod "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" (UID: "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:00.937897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937737    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:00.937897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937750    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:00.937897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.937762    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:00.939998 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.939967    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-podinfo" pod "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" (UID: "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:58:00.941136 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.941113    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-kube-api-access-zc6s9" pod "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" (UID: "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc"). InnerVolumeSpecName "kube-api-access-zc6s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:00.941230 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.941207    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-envoy" pod "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" (UID: "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:00.941269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:00.941240    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-token" pod "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" (UID: "e9889e75-036c-4358-a6c8-cbd4e7c0b0bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:01.038806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.038715    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc6s9\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-kube-api-access-zc6s9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:01.038806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.038741    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:01.038806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.038752    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:01.038806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.038761    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:01.038806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.038769    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:01.065240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.065210    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"]
Nov 02 14:58:01.081069 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.081038    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"]
Nov 02 14:58:01.245836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.245798    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"
Nov 02 14:58:01.245836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.245822    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm" event={"ID":"136896ee-fa2e-4c92-9f6e-f2b18f3ce509","Type":"ContainerDied","Data":"03255ffea6698f7104b2fcc8e1bdbf5a17d59bbb32ae8cd3451947d105d489a9"}
Nov 02 14:58:01.246075 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.245911    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03255ffea6698f7104b2fcc8e1bdbf5a17d59bbb32ae8cd3451947d105d489a9"
Nov 02 14:58:01.248131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.248103    5034 generic.go:356] "Generic (PLEG): container finished" podID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerID="78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734" exitCode=0
Nov 02 14:58:01.248282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.248138    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" event={"ID":"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc","Type":"ContainerDied","Data":"78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734"}
Nov 02 14:58:01.248282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.248157    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"
Nov 02 14:58:01.248282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.248179    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d" event={"ID":"e9889e75-036c-4358-a6c8-cbd4e7c0b0bc","Type":"ContainerDied","Data":"70a9e09570197883330d5ca5cc1e4b5465c2593d04e80f937e145da45918e7b2"}
Nov 02 14:58:01.248282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.248206    5034 scope.go:122] "RemoveContainer" containerID="78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734"
Nov 02 14:58:01.261026 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.261009    5034 scope.go:122] "RemoveContainer" containerID="0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e"
Nov 02 14:58:01.272310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.272284    5034 scope.go:122] "RemoveContainer" containerID="78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734"
Nov 02 14:58:01.272616 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:01.272590    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734\": container with ID starting with 78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734 not found: ID does not exist" containerID="78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734"
Nov 02 14:58:01.272716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.272627    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734"} err="failed to get container status \"78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734\": rpc error: code = NotFound desc = could not find container \"78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734\": container with ID starting with 78daa484dcb152e177720b1b38d8d924df47f7ba6e90f2d0ac5dfa33e7691734 not found: ID does not exist"
Nov 02 14:58:01.272716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.272651    5034 scope.go:122] "RemoveContainer" containerID="0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e"
Nov 02 14:58:01.272989 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:01.272963    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e\": container with ID starting with 0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e not found: ID does not exist" containerID="0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e"
Nov 02 14:58:01.273047 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.272999    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e"} err="failed to get container status \"0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e\": rpc error: code = NotFound desc = could not find container \"0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e\": container with ID starting with 0f76114871f4e4b090a956a20b447fd139733a9934b78259e3c984f1df324e8e not found: ID does not exist"
Nov 02 14:58:01.325164 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.325052    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"]
Nov 02 14:58:01.329135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.329108    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0d1ca7-openshift-default-54b7b5c5f9-7n68d"]
Nov 02 14:58:01.397596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:01.397567    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" path="/var/lib/kubelet/pods/e9889e75-036c-4358-a6c8-cbd4e7c0b0bc/volumes"
Nov 02 14:58:03.073351 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.073324    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:58:03.107684 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.107655    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:58:03.165345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.165318    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf25039-006c-447b-bbc7-a418d169eb39-kube-api-access-9c9qn\" (UniqueName: \"kubernetes.io/projected/eaf25039-006c-447b-bbc7-a418d169eb39-kube-api-access-9c9qn\") pod \"eaf25039-006c-447b-bbc7-a418d169eb39\" (UID: \"eaf25039-006c-447b-bbc7-a418d169eb39\") "
Nov 02 14:58:03.165488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.165418    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-plugins-config-volume\") pod \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\" (UID: \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\") "
Nov 02 14:58:03.165561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.165492    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-kube-api-access-wstvm\" (UniqueName: \"kubernetes.io/projected/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-kube-api-access-wstvm\") pod \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\" (UID: \"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc\") "
Nov 02 14:58:03.165615 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.165567    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf25039-006c-447b-bbc7-a418d169eb39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf25039-006c-447b-bbc7-a418d169eb39-plugins-config-volume\") pod \"eaf25039-006c-447b-bbc7-a418d169eb39\" (UID: \"eaf25039-006c-447b-bbc7-a418d169eb39\") "
Nov 02 14:58:03.166289 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.166267    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf25039-006c-447b-bbc7-a418d169eb39-plugins-config-volume" pod "eaf25039-006c-447b-bbc7-a418d169eb39" (UID: "eaf25039-006c-447b-bbc7-a418d169eb39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:03.167055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.166997    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-plugins-config-volume" pod "f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" (UID: "f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:03.168596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.168570    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"]
Nov 02 14:58:03.168693 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.168636    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf25039-006c-447b-bbc7-a418d169eb39-kube-api-access-9c9qn" pod "eaf25039-006c-447b-bbc7-a418d169eb39" (UID: "eaf25039-006c-447b-bbc7-a418d169eb39"). InnerVolumeSpecName "kube-api-access-9c9qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:03.169347 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.169325    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-kube-api-access-wstvm" pod "f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" (UID: "f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc"). InnerVolumeSpecName "kube-api-access-wstvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:03.173547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.173078    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"]
Nov 02 14:58:03.182029 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.182006    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"]
Nov 02 14:58:03.193290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.193267    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"]
Nov 02 14:58:03.265468 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.265428    5034 generic.go:356] "Generic (PLEG): container finished" podID="eaf25039-006c-447b-bbc7-a418d169eb39" containerID="7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81" exitCode=0
Nov 02 14:58:03.265647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.265522    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"
Nov 02 14:58:03.265647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.265482    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq" event={"ID":"eaf25039-006c-447b-bbc7-a418d169eb39","Type":"ContainerDied","Data":"7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81"}
Nov 02 14:58:03.265647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.265634    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq" event={"ID":"eaf25039-006c-447b-bbc7-a418d169eb39","Type":"ContainerDied","Data":"568fd54eb56d702e3f4f837667be11be5299462da048e1f38e2e08fc94231c94"}
Nov 02 14:58:03.266254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.265672    5034 scope.go:122] "RemoveContainer" containerID="7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81"
Nov 02 14:58:03.266931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.266910    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:03.266931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.266931    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstvm\" (UniqueName: \"kubernetes.io/projected/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc-kube-api-access-wstvm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:03.267149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.266941    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf25039-006c-447b-bbc7-a418d169eb39-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:03.267149 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.266953    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c9qn\" (UniqueName: \"kubernetes.io/projected/eaf25039-006c-447b-bbc7-a418d169eb39-kube-api-access-9c9qn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:03.273231 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.273207    5034 generic.go:356] "Generic (PLEG): container finished" podID="f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" containerID="5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22" exitCode=0
Nov 02 14:58:03.273358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.273249    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"
Nov 02 14:58:03.273358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.273285    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc" event={"ID":"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc","Type":"ContainerDied","Data":"5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22"}
Nov 02 14:58:03.273358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.273325    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc" event={"ID":"f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc","Type":"ContainerDied","Data":"42f7a93aa7735ab317912162409393c995010a326626792ca4f1c035180cfae2"}
Nov 02 14:58:03.286686 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.286667    5034 scope.go:122] "RemoveContainer" containerID="7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81"
Nov 02 14:58:03.287314 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:03.287288    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81\": container with ID starting with 7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81 not found: ID does not exist" containerID="7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81"
Nov 02 14:58:03.287479 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.287456    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81"} err="failed to get container status \"7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81\": rpc error: code = NotFound desc = could not find container \"7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81\": container with ID starting with 7df1e29db7225ad132b2583920f81aae7f3887d9f6e40d5cbadbb6f5d3677f81 not found: ID does not exist"
Nov 02 14:58:03.287595 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.287584    5034 scope.go:122] "RemoveContainer" containerID="5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22"
Nov 02 14:58:03.299874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.299856    5034 scope.go:122] "RemoveContainer" containerID="5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22"
Nov 02 14:58:03.300170 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:03.300146    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22\": container with ID starting with 5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22 not found: ID does not exist" containerID="5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22"
Nov 02 14:58:03.300221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.300179    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22"} err="failed to get container status \"5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22\": rpc error: code = NotFound desc = could not find container \"5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22\": container with ID starting with 5d137a4ccbb8af5238087a35a2b4740bddfcf512f271f80131da981fbbb3ca22 not found: ID does not exist"
Nov 02 14:58:03.407604 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.407576    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"]
Nov 02 14:58:03.414176 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.414151    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-960a68-67d8c5bdd9-jq6pc"]
Nov 02 14:58:03.450019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.449996    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"]
Nov 02 14:58:03.454938 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:03.454916    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-960a68-79f6558fdc-2wdvq"]
Nov 02 14:58:05.038483 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:05.038442    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"]
Nov 02 14:58:05.051192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:05.051165    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b059ebab-kz6wb"]
Nov 02 14:58:05.395300 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:05.395217    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae536c7e-d84e-4412-bdca-9f96647be30f" path="/var/lib/kubelet/pods/ae536c7e-d84e-4412-bdca-9f96647be30f/volumes"
Nov 02 14:58:05.395613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:05.395597    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf25039-006c-447b-bbc7-a418d169eb39" path="/var/lib/kubelet/pods/eaf25039-006c-447b-bbc7-a418d169eb39/volumes"
Nov 02 14:58:05.395989 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:05.395973    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" path="/var/lib/kubelet/pods/f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc/volumes"
Nov 02 14:58:07.353892 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.353859    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4" podUID="f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:07.466173 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.466137    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"]
Nov 02 14:58:07.499983 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.499946    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"]
Nov 02 14:58:07.972545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.972513    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"]
Nov 02 14:58:07.973050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973026    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy"
Nov 02 14:58:07.973050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973050    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973062    5034 cpu_manager.go:414] "containerMap: removing container" podUID="eaf25039-006c-447b-bbc7-a418d169eb39" containerName="payload-pre-processing"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973068    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf25039-006c-447b-bbc7-a418d169eb39" containerName="payload-pre-processing"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973080    5034 cpu_manager.go:414] "containerMap: removing container" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973085    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bd004e-2935-4b86-86ad-66d267777053" containerName="istio-proxy"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973091    5034 cpu_manager.go:414] "containerMap: removing container" podUID="136896ee-fa2e-4c92-9f6e-f2b18f3ce509" containerName="revoke-keys"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973097    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136896ee-fa2e-4c92-9f6e-f2b18f3ce509" containerName="revoke-keys"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973107    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973112    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973120    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973125    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973147    5034 cpu_manager.go:414] "containerMap: removing container" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973152    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f5b619-f69a-45ff-904a-d8507d2c2ea6" containerName="istio-proxy"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973170    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" containerName="payload-processing"
Nov 02 14:58:07.973179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973175    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" containerName="payload-processing"
Nov 02 14:58:07.973666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973250    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f567dd13-dc7c-4323-9c7e-cb7e9b3c8fcc" containerName="payload-processing"
Nov 02 14:58:07.973666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973262    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cab00b6-3287-4f34-a7e3-23fc811d615c" containerName="istio-proxy"
Nov 02 14:58:07.973666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973267    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1dc5d3-a4ad-42f7-9e88-fe4b3a4fe9d9" containerName="istio-proxy"
Nov 02 14:58:07.973666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973274    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf25039-006c-447b-bbc7-a418d169eb39" containerName="payload-pre-processing"
Nov 02 14:58:07.973666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973280    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="136896ee-fa2e-4c92-9f6e-f2b18f3ce509" containerName="revoke-keys"
Nov 02 14:58:07.973666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973287    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy"
Nov 02 14:58:07.973666 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.973292    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9889e75-036c-4358-a6c8-cbd4e7c0b0bc" containerName="istio-proxy"
Nov 02 14:58:07.978277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.978257    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:07.997882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:07.997602    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"]
Nov 02 14:58:08.006105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.005730    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:58:08.006105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.005820    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:58:08.006105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.005871    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:58:08.006105 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.005878    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:58:08.021129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.021097    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwtj\" (UniqueName: \"kubernetes.io/projected/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-kube-api-access-kqwtj\") pod \"maas-api-revoke-sub-4590b8df-nd5jv\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.021274 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.021144    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4590b8df-nd5jv\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.021350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.021322    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4590b8df-nd5jv\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.122233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.122198    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwtj\" (UniqueName: \"kubernetes.io/projected/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-kube-api-access-kqwtj\") pod \"maas-api-revoke-sub-4590b8df-nd5jv\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.122233 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.122238    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4590b8df-nd5jv\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.122476 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.122326    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4590b8df-nd5jv\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.123053 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.123032    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4590b8df-nd5jv\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.124733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.124710    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4590b8df-nd5jv\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.146208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.146176    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwtj\" (UniqueName: \"kubernetes.io/projected/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-kube-api-access-kqwtj\") pod \"maas-api-revoke-sub-4590b8df-nd5jv\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.293858 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.293820    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:08.603384 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:58:08.603352    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d3a8a6_4f39_4d48_9752_23a24b2a81a4.slice/crio-221f5bb8e976904e93019aa297d9aaacb743bd6c7083a67078d9fb79ae8f0709 WatchSource:0}: Error finding container 221f5bb8e976904e93019aa297d9aaacb743bd6c7083a67078d9fb79ae8f0709: Status 404 returned error can't find the container with id 221f5bb8e976904e93019aa297d9aaacb743bd6c7083a67078d9fb79ae8f0709
Nov 02 14:58:08.604251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:08.604224    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"]
Nov 02 14:58:09.326633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:09.326593    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv" event={"ID":"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4","Type":"ContainerStarted","Data":"0dfd6792719a7ff859ec2a6d5578f8c59fb747de8da9cc828260d831cc390dce"}
Nov 02 14:58:09.326866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:09.326658    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv" event={"ID":"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4","Type":"ContainerStarted","Data":"221f5bb8e976904e93019aa297d9aaacb743bd6c7083a67078d9fb79ae8f0709"}
Nov 02 14:58:09.373391 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:09.373332    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv" podStartSLOduration=2.373311974 podStartE2EDuration="2.373311974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:09.368257912 +0000 UTC m=+4480.119562351" watchObservedRunningTime="2026-11-02 14:58:09.373311974 +0000 UTC m=+4480.124616415"
Nov 02 14:58:09.952377 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:09.952338    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"]
Nov 02 14:58:09.981773 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:09.981738    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"]
Nov 02 14:58:10.298158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.298116    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:58:10.335913 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.335882    5034 generic.go:356] "Generic (PLEG): container finished" podID="a2d3a8a6-4f39-4d48-9752-23a24b2a81a4" containerID="0dfd6792719a7ff859ec2a6d5578f8c59fb747de8da9cc828260d831cc390dce" exitCode=0
Nov 02 14:58:10.336087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.335967    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv" event={"ID":"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4","Type":"ContainerDied","Data":"0dfd6792719a7ff859ec2a6d5578f8c59fb747de8da9cc828260d831cc390dce"}
Nov 02 14:58:10.337495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.337471    5034 generic.go:356] "Generic (PLEG): container finished" podID="f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" containerID="cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242" exitCode=0
Nov 02 14:58:10.337613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.337518    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"
Nov 02 14:58:10.337613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.337558    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4" event={"ID":"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938","Type":"ContainerDied","Data":"cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242"}
Nov 02 14:58:10.337613 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.337612    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4" event={"ID":"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938","Type":"ContainerDied","Data":"7eb4cff1ec5bf49b316650d5bbefbc2b8368eb005fff7be173fd2d8088821a61"}
Nov 02 14:58:10.337860 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.337631    5034 scope.go:122] "RemoveContainer" containerID="cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242"
Nov 02 14:58:10.350730 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.350706    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-kube-api-access-rb8sp\" (UniqueName: \"kubernetes.io/projected/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-kube-api-access-rb8sp\") pod \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\" (UID: \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\") "
Nov 02 14:58:10.350918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.350768    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-plugins-config-volume\") pod \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\" (UID: \"f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938\") "
Nov 02 14:58:10.351507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.351485    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-plugins-config-volume" pod "f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" (UID: "f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:10.360737 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.360701    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-kube-api-access-rb8sp" pod "f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" (UID: "f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938"). InnerVolumeSpecName "kube-api-access-rb8sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:10.363671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.363651    5034 scope.go:122] "RemoveContainer" containerID="cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242"
Nov 02 14:58:10.364056 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:10.364015    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242\": container with ID starting with cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242 not found: ID does not exist" containerID="cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242"
Nov 02 14:58:10.364198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.364065    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242"} err="failed to get container status \"cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242\": rpc error: code = NotFound desc = could not find container \"cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242\": container with ID starting with cfd2ca081904e5be53b60603252603283a42da8abdf7cf71504c3fd764b08242 not found: ID does not exist"
Nov 02 14:58:10.452761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.452717    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8sp\" (UniqueName: \"kubernetes.io/projected/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-kube-api-access-rb8sp\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:10.452761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.452761    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:10.697633 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.697607    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"]
Nov 02 14:58:10.702632 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:10.702602    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0d1ca7-7985db4f5c-bb8d4"]
Nov 02 14:58:11.221988 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.221956    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"]
Nov 02 14:58:11.254180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.254150    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"]
Nov 02 14:58:11.396894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.396859    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" path="/var/lib/kubelet/pods/f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938/volumes"
Nov 02 14:58:11.540646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.540623    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:11.666704 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.666678    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-openshift-ca\") pod \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") "
Nov 02 14:58:11.666893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.666798    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-xks-ca\") pod \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") "
Nov 02 14:58:11.666893 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.666827    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-kube-api-access-kqwtj\" (UniqueName: \"kubernetes.io/projected/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-kube-api-access-kqwtj\") pod \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\" (UID: \"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4\") "
Nov 02 14:58:11.667033 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.667007    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-openshift-ca" pod "a2d3a8a6-4f39-4d48-9752-23a24b2a81a4" (UID: "a2d3a8a6-4f39-4d48-9752-23a24b2a81a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:11.667259 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.667236    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:11.669031 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.669006    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-kube-api-access-kqwtj" pod "a2d3a8a6-4f39-4d48-9752-23a24b2a81a4" (UID: "a2d3a8a6-4f39-4d48-9752-23a24b2a81a4"). InnerVolumeSpecName "kube-api-access-kqwtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:11.669131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.669080    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-xks-ca" pod "a2d3a8a6-4f39-4d48-9752-23a24b2a81a4" (UID: "a2d3a8a6-4f39-4d48-9752-23a24b2a81a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:11.768760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.768693    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:11.768760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:11.768721    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqwtj\" (UniqueName: \"kubernetes.io/projected/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4-kube-api-access-kqwtj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:12.378683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:12.378651    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv" event={"ID":"a2d3a8a6-4f39-4d48-9752-23a24b2a81a4","Type":"ContainerDied","Data":"221f5bb8e976904e93019aa297d9aaacb743bd6c7083a67078d9fb79ae8f0709"}
Nov 02 14:58:12.378683 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:12.378685    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"
Nov 02 14:58:12.379182 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:12.378701    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="221f5bb8e976904e93019aa297d9aaacb743bd6c7083a67078d9fb79ae8f0709"
Nov 02 14:58:12.467169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:12.467134    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv" podUID="39520cdc-cdca-4c22-81ef-480183b2d652" containerName="payload-processing" containerID="cri-o://58c9471215efcafc5c11fa233163667d1cc42e84a0d68993aa12dd3311f624e8" gracePeriod=25
Nov 02 14:58:12.500854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:12.500818    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj" podUID="04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" containerName="payload-pre-processing" containerID="cri-o://a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380" gracePeriod=25
Nov 02 14:58:13.040156 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:13.040122    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv" podUID="39520cdc-cdca-4c22-81ef-480183b2d652" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:13.040571 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:13.040548    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj" podUID="04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:14.952833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:14.952768    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr" podUID="ecd91416-7106-4f87-80d6-3d283e12c1a5" containerName="payload-processing" containerID="cri-o://7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158" gracePeriod=25
Nov 02 14:58:14.983282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:14.983244    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m" podUID="89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" containerName="payload-pre-processing" containerID="cri-o://ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67" gracePeriod=25
Nov 02 14:58:15.031360 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:15.031327    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"]
Nov 02 14:58:15.042271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:15.042246    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87fdb2d5-gf292"]
Nov 02 14:58:15.394910 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:15.394878    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4db072d-3be7-4a1a-9357-e06c49abc2c0" path="/var/lib/kubelet/pods/b4db072d-3be7-4a1a-9357-e06c49abc2c0/volumes"
Nov 02 14:58:16.094315 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.094273    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main" containerID="cri-o://c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588" gracePeriod=45
Nov 02 14:58:16.111528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.111493    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" containerID="cri-o://c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb" gracePeriod=45
Nov 02 14:58:16.222229 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.222174    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw" podUID="2358bb25-5f5b-42a4-9678-852be7940e34" containerName="payload-processing" containerID="cri-o://bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685" gracePeriod=25
Nov 02 14:58:16.254589 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.254397    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q" podUID="e28fcc5d-ad6f-426e-a370-5f6576bf37d6" containerName="payload-pre-processing" containerID="cri-o://117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab" gracePeriod=25
Nov 02 14:58:16.400986 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.400962    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:58:16.404432 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.404409    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:58:16.408307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.408281    5034 generic.go:356] "Generic (PLEG): container finished" podID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerID="c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb" exitCode=0
Nov 02 14:58:16.408397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.408322    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"
Nov 02 14:58:16.408397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.408367    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" event={"ID":"903cf918-c71c-42c4-8fb4-12ebd76d9b6f","Type":"ContainerDied","Data":"c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb"}
Nov 02 14:58:16.408495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.408410    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" event={"ID":"903cf918-c71c-42c4-8fb4-12ebd76d9b6f","Type":"ContainerDied","Data":"dacf527616de266e230d41ad413e8aaf46e7e9299d3ec35f77f30c974b9d9a28"}
Nov 02 14:58:16.408495 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.408430    5034 scope.go:122] "RemoveContainer" containerID="c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb"
Nov 02 14:58:16.409984 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.409964    5034 generic.go:356] "Generic (PLEG): container finished" podID="08a21e6a-28af-4b2e-9251-32f455208189" containerID="c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588" exitCode=0
Nov 02 14:58:16.410089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.409999    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"
Nov 02 14:58:16.410089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.410033    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" event={"ID":"08a21e6a-28af-4b2e-9251-32f455208189","Type":"ContainerDied","Data":"c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588"}
Nov 02 14:58:16.410089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.410061    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md" event={"ID":"08a21e6a-28af-4b2e-9251-32f455208189","Type":"ContainerDied","Data":"76e76df2e7c2812b5023dc40b9d748df64b19870b9d003f8c3eb1550f0b39ee3"}
Nov 02 14:58:16.421660 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.421632    5034 scope.go:122] "RemoveContainer" containerID="c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb"
Nov 02 14:58:16.422169 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:16.422142    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb\": container with ID starting with c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb not found: ID does not exist" containerID="c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb"
Nov 02 14:58:16.422258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.422214    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb"} err="failed to get container status \"c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb\": rpc error: code = NotFound desc = could not find container \"c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb\": container with ID starting with c310a6e78e55cf4e25a7b5a9a9196c1acc2c2a3ff1e0d5863a6ca262c619bdeb not found: ID does not exist"
Nov 02 14:58:16.422258 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.422239    5034 scope.go:122] "RemoveContainer" containerID="c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588"
Nov 02 14:58:16.433059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.433038    5034 scope.go:122] "RemoveContainer" containerID="c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588"
Nov 02 14:58:16.433344 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:16.433323    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588\": container with ID starting with c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588 not found: ID does not exist" containerID="c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588"
Nov 02 14:58:16.433425 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.433351    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588"} err="failed to get container status \"c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588\": rpc error: code = NotFound desc = could not find container \"c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588\": container with ID starting with c2f750a3f22312af2cdeb63966744d99d7b20fdb0dabc0b8f736a99872465588 not found: ID does not exist"
Nov 02 14:58:16.520733 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.520703    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tls-certs\") pod \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.520756    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-kube-api-access-22xsb\" (UniqueName: \"kubernetes.io/projected/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-kube-api-access-22xsb\") pod \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.520823    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08a21e6a-28af-4b2e-9251-32f455208189-kube-api-access-rmc2x\" (UniqueName: \"kubernetes.io/projected/08a21e6a-28af-4b2e-9251-32f455208189-kube-api-access-rmc2x\") pod \"08a21e6a-28af-4b2e-9251-32f455208189\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.520864    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08a21e6a-28af-4b2e-9251-32f455208189-tls-certs\" (UniqueName: \"kubernetes.io/secret/08a21e6a-28af-4b2e-9251-32f455208189-tls-certs\") pod \"08a21e6a-28af-4b2e-9251-32f455208189\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.520913    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-tmp-dir\") pod \"08a21e6a-28af-4b2e-9251-32f455208189\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.520965    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-dshm\") pod \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521020    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-dshm\") pod \"08a21e6a-28af-4b2e-9251-32f455208189\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521110    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-model-cache\") pod \"08a21e6a-28af-4b2e-9251-32f455208189\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521157    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-model-cache\") pod \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521184    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tmp-dir\") pod \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") "
Nov 02 14:58:16.521262 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521250    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-tmp-dir" pod "08a21e6a-28af-4b2e-9251-32f455208189" (UID: "08a21e6a-28af-4b2e-9251-32f455208189"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:16.521980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521287    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-home\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-home\") pod \"08a21e6a-28af-4b2e-9251-32f455208189\" (UID: \"08a21e6a-28af-4b2e-9251-32f455208189\") "
Nov 02 14:58:16.521980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521317    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-home\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-home\") pod \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\" (UID: \"903cf918-c71c-42c4-8fb4-12ebd76d9b6f\") "
Nov 02 14:58:16.521980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521475    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-model-cache" pod "08a21e6a-28af-4b2e-9251-32f455208189" (UID: "08a21e6a-28af-4b2e-9251-32f455208189"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:16.521980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521869    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tmp-dir" pod "903cf918-c71c-42c4-8fb4-12ebd76d9b6f" (UID: "903cf918-c71c-42c4-8fb4-12ebd76d9b6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:16.521980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521916    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.521980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521941    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.521980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521941    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-model-cache" pod "903cf918-c71c-42c4-8fb4-12ebd76d9b6f" (UID: "903cf918-c71c-42c4-8fb4-12ebd76d9b6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:16.521980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.521972    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-home" pod "903cf918-c71c-42c4-8fb4-12ebd76d9b6f" (UID: "903cf918-c71c-42c4-8fb4-12ebd76d9b6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:16.522908 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.522092    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-home" pod "08a21e6a-28af-4b2e-9251-32f455208189" (UID: "08a21e6a-28af-4b2e-9251-32f455208189"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:16.524055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.523760    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a21e6a-28af-4b2e-9251-32f455208189-tls-certs" pod "08a21e6a-28af-4b2e-9251-32f455208189" (UID: "08a21e6a-28af-4b2e-9251-32f455208189"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:16.524055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.523916    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a21e6a-28af-4b2e-9251-32f455208189-kube-api-access-rmc2x" pod "08a21e6a-28af-4b2e-9251-32f455208189" (UID: "08a21e6a-28af-4b2e-9251-32f455208189"). InnerVolumeSpecName "kube-api-access-rmc2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:16.524055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.523931    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tls-certs" pod "903cf918-c71c-42c4-8fb4-12ebd76d9b6f" (UID: "903cf918-c71c-42c4-8fb4-12ebd76d9b6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:16.525407 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.525381    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-kube-api-access-22xsb" pod "903cf918-c71c-42c4-8fb4-12ebd76d9b6f" (UID: "903cf918-c71c-42c4-8fb4-12ebd76d9b6f"). InnerVolumeSpecName "kube-api-access-22xsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:16.525497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.525451    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-dshm" pod "903cf918-c71c-42c4-8fb4-12ebd76d9b6f" (UID: "903cf918-c71c-42c4-8fb4-12ebd76d9b6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:16.525497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.525461    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-dshm" pod "08a21e6a-28af-4b2e-9251-32f455208189" (UID: "08a21e6a-28af-4b2e-9251-32f455208189"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:16.622874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622769    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.622874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622816    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.622874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622829    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.622874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622840    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.622874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622850    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.622874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622862    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22xsb\" (UniqueName: \"kubernetes.io/projected/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-kube-api-access-22xsb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.622874 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622877    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmc2x\" (UniqueName: \"kubernetes.io/projected/08a21e6a-28af-4b2e-9251-32f455208189-kube-api-access-rmc2x\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.623235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622889    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08a21e6a-28af-4b2e-9251-32f455208189-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.623235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622902    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903cf918-c71c-42c4-8fb4-12ebd76d9b6f-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.623235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.622913    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08a21e6a-28af-4b2e-9251-32f455208189-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:16.790477 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.790445    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"]
Nov 02 14:58:16.801096 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.801067    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s"]
Nov 02 14:58:16.840792 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.840756    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"]
Nov 02 14:58:16.845139 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:16.845115    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-92f249/distinct-2-w1-92f249-kserve-bcc58ff59-dc4md"]
Nov 02 14:58:17.333502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:17.333460    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-92f249/premium-w1-92f249-kserve-767f74bf5c-lvl6s" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:58:17.394014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:17.393933    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08a21e6a-28af-4b2e-9251-32f455208189" path="/var/lib/kubelet/pods/08a21e6a-28af-4b2e-9251-32f455208189/volumes"
Nov 02 14:58:17.394466 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:17.394445    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" path="/var/lib/kubelet/pods/903cf918-c71c-42c4-8fb4-12ebd76d9b6f/volumes"
Nov 02 14:58:18.197294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.197245    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" containerID="cri-o://59da64a98b1eb70ae7857d1ad90234ec8480fdb0da26c219c62acf9cfc0c641c" gracePeriod=45
Nov 02 14:58:18.199114 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.199081    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main" containerID="cri-o://72fe0e3de55b7fbd25af30a4a2da4f3948057cc503e3655b9a04b7512024da8f" gracePeriod=45
Nov 02 14:58:18.214548 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.214515    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main" containerID="cri-o://2eedf28857326edb415485db95aa14595d94103f72390c4574d2eeaf03cc6ec4" gracePeriod=45
Nov 02 14:58:18.218401 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.218370    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main" containerID="cri-o://8c50f3514fc4dae2351193feaa07abfc5b23bf0df630d425b99390d59258c47d" gracePeriod=45
Nov 02 14:58:18.334048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.334012    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": dial tcp 10.132.0.103:8000: connect: connection refused"
Nov 02 14:58:18.442403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.438138    5034 generic.go:356] "Generic (PLEG): container finished" podID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerID="2eedf28857326edb415485db95aa14595d94103f72390c4574d2eeaf03cc6ec4" exitCode=0
Nov 02 14:58:18.442403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.438268    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" event={"ID":"5b39766e-2b9e-4941-bf05-9ea55c506925","Type":"ContainerDied","Data":"2eedf28857326edb415485db95aa14595d94103f72390c4574d2eeaf03cc6ec4"}
Nov 02 14:58:18.442403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.440742    5034 generic.go:356] "Generic (PLEG): container finished" podID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerID="8c50f3514fc4dae2351193feaa07abfc5b23bf0df630d425b99390d59258c47d" exitCode=0
Nov 02 14:58:18.442403 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.440907    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" event={"ID":"b2c0415b-c21f-4867-80cc-5bb953cb0532","Type":"ContainerDied","Data":"8c50f3514fc4dae2351193feaa07abfc5b23bf0df630d425b99390d59258c47d"}
Nov 02 14:58:18.443544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.443389    5034 generic.go:356] "Generic (PLEG): container finished" podID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerID="72fe0e3de55b7fbd25af30a4a2da4f3948057cc503e3655b9a04b7512024da8f" exitCode=0
Nov 02 14:58:18.443544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.443493    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" event={"ID":"e06878a8-2080-4a80-a243-d37ec2ed7076","Type":"ContainerDied","Data":"72fe0e3de55b7fbd25af30a4a2da4f3948057cc503e3655b9a04b7512024da8f"}
Nov 02 14:58:18.463264 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:18.459038    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:58:18.463264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.459295    5034 generic.go:356] "Generic (PLEG): container finished" podID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerID="59da64a98b1eb70ae7857d1ad90234ec8480fdb0da26c219c62acf9cfc0c641c" exitCode=0
Nov 02 14:58:18.463264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.459389    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" event={"ID":"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1","Type":"ContainerDied","Data":"59da64a98b1eb70ae7857d1ad90234ec8480fdb0da26c219c62acf9cfc0c641c"}
Nov 02 14:58:18.463264 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.460760    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr" podUID="ecd91416-7106-4f87-80d6-3d283e12c1a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:18.464282 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.464212    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m" podUID="89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:18.485000 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.484976    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"]
Nov 02 14:58:18.485236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.485213    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy" containerID="cri-o://3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77" gracePeriod=30
Nov 02 14:58:18.523232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.523156    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:58:18.546966 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.546941    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:58:18.641273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.641247    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:58:18.643671 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.643647    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tmp-dir\") pod \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") "
Nov 02 14:58:18.643825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.643685    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-model-cache\") pod \"5b39766e-2b9e-4941-bf05-9ea55c506925\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") "
Nov 02 14:58:18.643825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.643742    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-kube-api-access-wnbl9\" (UniqueName: \"kubernetes.io/projected/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-kube-api-access-wnbl9\") pod \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") "
Nov 02 14:58:18.643825 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.643814    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-dshm\") pod \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") "
Nov 02 14:58:18.643943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.643849    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b39766e-2b9e-4941-bf05-9ea55c506925-kube-api-access-dcpv8\" (UniqueName: \"kubernetes.io/projected/5b39766e-2b9e-4941-bf05-9ea55c506925-kube-api-access-dcpv8\") pod \"5b39766e-2b9e-4941-bf05-9ea55c506925\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") "
Nov 02 14:58:18.643943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.643878    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-tmp-dir\") pod \"5b39766e-2b9e-4941-bf05-9ea55c506925\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") "
Nov 02 14:58:18.643943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.643925    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tmp-dir" pod "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" (UID: "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.644037 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.643944    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-home\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-home\") pod \"5b39766e-2b9e-4941-bf05-9ea55c506925\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") "
Nov 02 14:58:18.644078 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.644066    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-dshm\") pod \"5b39766e-2b9e-4941-bf05-9ea55c506925\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") "
Nov 02 14:58:18.644140 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.644123    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b39766e-2b9e-4941-bf05-9ea55c506925-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b39766e-2b9e-4941-bf05-9ea55c506925-tls-certs\") pod \"5b39766e-2b9e-4941-bf05-9ea55c506925\" (UID: \"5b39766e-2b9e-4941-bf05-9ea55c506925\") "
Nov 02 14:58:18.644174 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.644159    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-home\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-home\") pod \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") "
Nov 02 14:58:18.644222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.644194    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tls-certs\") pod \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") "
Nov 02 14:58:18.644261 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.644249    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-model-cache\") pod \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\" (UID: \"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1\") "
Nov 02 14:58:18.644324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.644308    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:58:18.644710 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.644687    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.644918 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.644896    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-model-cache" pod "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" (UID: "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.644993 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.644904    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-model-cache" pod "5b39766e-2b9e-4941-bf05-9ea55c506925" (UID: "5b39766e-2b9e-4941-bf05-9ea55c506925"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.645793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.645443    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-home" pod "5b39766e-2b9e-4941-bf05-9ea55c506925" (UID: "5b39766e-2b9e-4941-bf05-9ea55c506925"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.645793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.645644    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-tmp-dir" pod "5b39766e-2b9e-4941-bf05-9ea55c506925" (UID: "5b39766e-2b9e-4941-bf05-9ea55c506925"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.645981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.645916    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-home" pod "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" (UID: "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.646199 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.646179    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-dshm" pod "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" (UID: "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.647288 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.647264    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b39766e-2b9e-4941-bf05-9ea55c506925-tls-certs" pod "5b39766e-2b9e-4941-bf05-9ea55c506925" (UID: "5b39766e-2b9e-4941-bf05-9ea55c506925"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:18.647475 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.647459    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b39766e-2b9e-4941-bf05-9ea55c506925-kube-api-access-dcpv8" pod "5b39766e-2b9e-4941-bf05-9ea55c506925" (UID: "5b39766e-2b9e-4941-bf05-9ea55c506925"). InnerVolumeSpecName "kube-api-access-dcpv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:18.648124 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.648101    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-dshm" pod "5b39766e-2b9e-4941-bf05-9ea55c506925" (UID: "5b39766e-2b9e-4941-bf05-9ea55c506925"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.648208 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.648142    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-kube-api-access-wnbl9" pod "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" (UID: "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1"). InnerVolumeSpecName "kube-api-access-wnbl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:18.649257 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.649222    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tls-certs" pod "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" (UID: "bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:18.745148 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745116    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06878a8-2080-4a80-a243-d37ec2ed7076-tls-certs\" (UniqueName: \"kubernetes.io/secret/e06878a8-2080-4a80-a243-d37ec2ed7076-tls-certs\") pod \"e06878a8-2080-4a80-a243-d37ec2ed7076\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") "
Nov 02 14:58:18.745321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745163    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-home\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-home\") pod \"b2c0415b-c21f-4867-80cc-5bb953cb0532\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") "
Nov 02 14:58:18.745321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745229    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-tmp-dir\") pod \"e06878a8-2080-4a80-a243-d37ec2ed7076\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") "
Nov 02 14:58:18.745321 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745266    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-dshm\") pod \"b2c0415b-c21f-4867-80cc-5bb953cb0532\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") "
Nov 02 14:58:18.745565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745339    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-tmp-dir\") pod \"b2c0415b-c21f-4867-80cc-5bb953cb0532\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") "
Nov 02 14:58:18.745565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745366    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c0415b-c21f-4867-80cc-5bb953cb0532-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c0415b-c21f-4867-80cc-5bb953cb0532-tls-certs\") pod \"b2c0415b-c21f-4867-80cc-5bb953cb0532\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") "
Nov 02 14:58:18.745565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745407    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-dshm\") pod \"e06878a8-2080-4a80-a243-d37ec2ed7076\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") "
Nov 02 14:58:18.745565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745448    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06878a8-2080-4a80-a243-d37ec2ed7076-kube-api-access-xng2w\" (UniqueName: \"kubernetes.io/projected/e06878a8-2080-4a80-a243-d37ec2ed7076-kube-api-access-xng2w\") pod \"e06878a8-2080-4a80-a243-d37ec2ed7076\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") "
Nov 02 14:58:18.745565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745478    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-model-cache\") pod \"b2c0415b-c21f-4867-80cc-5bb953cb0532\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") "
Nov 02 14:58:18.745565 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745496    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-home" pod "b2c0415b-c21f-4867-80cc-5bb953cb0532" (UID: "b2c0415b-c21f-4867-80cc-5bb953cb0532"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.745907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745586    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-model-cache\") pod \"e06878a8-2080-4a80-a243-d37ec2ed7076\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") "
Nov 02 14:58:18.745907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745638    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-home\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-home\") pod \"e06878a8-2080-4a80-a243-d37ec2ed7076\" (UID: \"e06878a8-2080-4a80-a243-d37ec2ed7076\") "
Nov 02 14:58:18.745907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745738    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c0415b-c21f-4867-80cc-5bb953cb0532-kube-api-access-2xnmx\" (UniqueName: \"kubernetes.io/projected/b2c0415b-c21f-4867-80cc-5bb953cb0532-kube-api-access-2xnmx\") pod \"b2c0415b-c21f-4867-80cc-5bb953cb0532\" (UID: \"b2c0415b-c21f-4867-80cc-5bb953cb0532\") "
Nov 02 14:58:18.745907 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.745746    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-model-cache" pod "b2c0415b-c21f-4867-80cc-5bb953cb0532" (UID: "b2c0415b-c21f-4867-80cc-5bb953cb0532"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.746307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746244    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbl9\" (UniqueName: \"kubernetes.io/projected/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-kube-api-access-wnbl9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746307 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746302    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746318    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcpv8\" (UniqueName: \"kubernetes.io/projected/5b39766e-2b9e-4941-bf05-9ea55c506925-kube-api-access-dcpv8\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746334    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746332    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-model-cache" pod "e06878a8-2080-4a80-a243-d37ec2ed7076" (UID: "e06878a8-2080-4a80-a243-d37ec2ed7076"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746347    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746359    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746373    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746387    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b39766e-2b9e-4941-bf05-9ea55c506925-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746400    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746412    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746425    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746437    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746449    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b39766e-2b9e-4941-bf05-9ea55c506925-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.746506 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746505    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-tmp-dir" pod "b2c0415b-c21f-4867-80cc-5bb953cb0532" (UID: "b2c0415b-c21f-4867-80cc-5bb953cb0532"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.747209 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.746687    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-tmp-dir" pod "e06878a8-2080-4a80-a243-d37ec2ed7076" (UID: "e06878a8-2080-4a80-a243-d37ec2ed7076"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.747450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.747428    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-home" pod "e06878a8-2080-4a80-a243-d37ec2ed7076" (UID: "e06878a8-2080-4a80-a243-d37ec2ed7076"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.748221 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.748193    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06878a8-2080-4a80-a243-d37ec2ed7076-tls-certs" pod "e06878a8-2080-4a80-a243-d37ec2ed7076" (UID: "e06878a8-2080-4a80-a243-d37ec2ed7076"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:18.749022 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.748993    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c0415b-c21f-4867-80cc-5bb953cb0532-tls-certs" pod "b2c0415b-c21f-4867-80cc-5bb953cb0532" (UID: "b2c0415b-c21f-4867-80cc-5bb953cb0532"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:18.749676 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.749645    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-dshm" pod "b2c0415b-c21f-4867-80cc-5bb953cb0532" (UID: "b2c0415b-c21f-4867-80cc-5bb953cb0532"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.749764 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.749729    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-dshm" pod "e06878a8-2080-4a80-a243-d37ec2ed7076" (UID: "e06878a8-2080-4a80-a243-d37ec2ed7076"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:18.749963 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.749945    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06878a8-2080-4a80-a243-d37ec2ed7076-kube-api-access-xng2w" pod "e06878a8-2080-4a80-a243-d37ec2ed7076" (UID: "e06878a8-2080-4a80-a243-d37ec2ed7076"). InnerVolumeSpecName "kube-api-access-xng2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:18.751275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.751255    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c0415b-c21f-4867-80cc-5bb953cb0532-kube-api-access-2xnmx" pod "b2c0415b-c21f-4867-80cc-5bb953cb0532" (UID: "b2c0415b-c21f-4867-80cc-5bb953cb0532"). InnerVolumeSpecName "kube-api-access-2xnmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:18.847995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.847967    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.847995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.847994    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xnmx\" (UniqueName: \"kubernetes.io/projected/b2c0415b-c21f-4867-80cc-5bb953cb0532-kube-api-access-2xnmx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.848196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.848004    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e06878a8-2080-4a80-a243-d37ec2ed7076-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.848196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.848014    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.848196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.848022    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.848196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.848029    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c0415b-c21f-4867-80cc-5bb953cb0532-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.848196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.848036    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c0415b-c21f-4867-80cc-5bb953cb0532-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.848196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.848044    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.848196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.848051    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xng2w\" (UniqueName: \"kubernetes.io/projected/e06878a8-2080-4a80-a243-d37ec2ed7076-kube-api-access-xng2w\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.848196 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.848059    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e06878a8-2080-4a80-a243-d37ec2ed7076-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:18.990460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:18.990426    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q" podUID="e28fcc5d-ad6f-426e-a370-5f6576bf37d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:19.454087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454062    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"]
Nov 02 14:58:19.454563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454549    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" containerName="payload-processing"
Nov 02 14:58:19.454563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454563    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" containerName="payload-processing"
Nov 02 14:58:19.454563 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454576    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454582    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454592    5034 cpu_manager.go:414] "containerMap: removing container" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454599    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454622    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454627    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454638    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454642    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454653    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a2d3a8a6-4f39-4d48-9752-23a24b2a81a4" containerName="revoke-keys"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454659    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d3a8a6-4f39-4d48-9752-23a24b2a81a4" containerName="revoke-keys"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454672    5034 cpu_manager.go:414] "containerMap: removing container" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454677    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454688    5034 cpu_manager.go:414] "containerMap: removing container" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454693    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454759    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454770    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" containerName="main"
Nov 02 14:58:19.454793 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454804    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d3a8a6-4f39-4d48-9752-23a24b2a81a4" containerName="revoke-keys"
Nov 02 14:58:19.455402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454813    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="08a21e6a-28af-4b2e-9251-32f455208189" containerName="main"
Nov 02 14:58:19.455402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454820    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="903cf918-c71c-42c4-8fb4-12ebd76d9b6f" containerName="main"
Nov 02 14:58:19.455402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454827    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f3d6f9-5cb7-44ea-8a61-1d7e9faf6938" containerName="payload-processing"
Nov 02 14:58:19.455402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454834    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" containerName="main"
Nov 02 14:58:19.455402 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.454838    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" containerName="main"
Nov 02 14:58:19.459482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.459461    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.467499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.467474    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"
Nov 02 14:58:19.467499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.467486    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm" event={"ID":"e06878a8-2080-4a80-a243-d37ec2ed7076","Type":"ContainerDied","Data":"8c9524d350aebf32034509752f6b14b817feeec161bdf3e21a118a654a8e8494"}
Nov 02 14:58:19.467667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.467533    5034 scope.go:122] "RemoveContainer" containerID="72fe0e3de55b7fbd25af30a4a2da4f3948057cc503e3655b9a04b7512024da8f"
Nov 02 14:58:19.469014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.468992    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"
Nov 02 14:58:19.469127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.469045    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n" event={"ID":"bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1","Type":"ContainerDied","Data":"c39c281994c35ed6d1c16519b4e109cb53dfc0aed1e03f00e0cf438412b5fd2c"}
Nov 02 14:58:19.474449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.473913    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"
Nov 02 14:58:19.474449 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.474297    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk" event={"ID":"5b39766e-2b9e-4941-bf05-9ea55c506925","Type":"ContainerDied","Data":"125552a4ef5bde955b72db76bbeeb47b0fde4c403612ee8a30ac3596c0564815"}
Nov 02 14:58:19.475980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.475033    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:58:19.475980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.475919    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:58:19.475980 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.475945    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:58:19.476203 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.476044    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:58:19.477533 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.477507    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5" event={"ID":"b2c0415b-c21f-4867-80cc-5bb953cb0532","Type":"ContainerDied","Data":"ca382985524d9230d515869b21d112fd6849e0b45b6829914cb56c7325269461"}
Nov 02 14:58:19.477770 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.477752    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"
Nov 02 14:58:19.479395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.479158    5034 scope.go:122] "RemoveContainer" containerID="59da64a98b1eb70ae7857d1ad90234ec8480fdb0da26c219c62acf9cfc0c641c"
Nov 02 14:58:19.482345 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.482322    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"]
Nov 02 14:58:19.496999 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.496976    5034 scope.go:122] "RemoveContainer" containerID="2eedf28857326edb415485db95aa14595d94103f72390c4574d2eeaf03cc6ec4"
Nov 02 14:58:19.512118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.510626    5034 scope.go:122] "RemoveContainer" containerID="8c50f3514fc4dae2351193feaa07abfc5b23bf0df630d425b99390d59258c47d"
Nov 02 14:58:19.541855 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.541832    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"]
Nov 02 14:58:19.546430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.546405    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-2-w4-ad42e0-kserve-8cdd5798d-t77tm"]
Nov 02 14:58:19.555978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.555955    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d601ecfb-mrg9n\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.556079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.556014    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d601ecfb-mrg9n\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.556130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.556093    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnd6m\" (UniqueName: \"kubernetes.io/projected/896ac00f-5537-4e47-9356-c88ef5e1fbc4-kube-api-access-rnd6m\") pod \"maas-api-revoke-sub-d601ecfb-mrg9n\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.581691 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.581659    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"]
Nov 02 14:58:19.584397 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.584370    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/distinct-w4-ad42e0-kserve-7665748d55-lwg7n"]
Nov 02 14:58:19.621359 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.621330    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"]
Nov 02 14:58:19.624014 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.623982    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/simulator-w4-ad42e0-kserve-6bdd8f7d85-pxpdk"]
Nov 02 14:58:19.656922 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.656894    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d601ecfb-mrg9n\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.657020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.656972    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d601ecfb-mrg9n\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.657020 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.656996    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnd6m\" (UniqueName: \"kubernetes.io/projected/896ac00f-5537-4e47-9356-c88ef5e1fbc4-kube-api-access-rnd6m\") pod \"maas-api-revoke-sub-d601ecfb-mrg9n\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.657645 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.657623    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d601ecfb-mrg9n\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.659215 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.659194    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d601ecfb-mrg9n\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.659505 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.659486    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"]
Nov 02 14:58:19.664902 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.664884    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-ad42e0/premium-w4-ad42e0-kserve-7bb564bc8d-8jgk5"]
Nov 02 14:58:19.681534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.681505    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnd6m\" (UniqueName: \"kubernetes.io/projected/896ac00f-5537-4e47-9356-c88ef5e1fbc4-kube-api-access-rnd6m\") pod \"maas-api-revoke-sub-d601ecfb-mrg9n\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:19.773843 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:19.773814    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:20.004370 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:20.004326    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw" podUID="2358bb25-5f5b-42a4-9678-852be7940e34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:20.082826 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:58:20.082750    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896ac00f_5537_4e47_9356_c88ef5e1fbc4.slice/crio-bafdfdb7b687dffb572a6b55450e488064d56abc90f97048d287f1466d402ce2 WatchSource:0}: Error finding container bafdfdb7b687dffb572a6b55450e488064d56abc90f97048d287f1466d402ce2: Status 404 returned error can't find the container with id bafdfdb7b687dffb572a6b55450e488064d56abc90f97048d287f1466d402ce2
Nov 02 14:58:20.083088 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:20.083062    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"]
Nov 02 14:58:20.494529 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:20.494487    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n" event={"ID":"896ac00f-5537-4e47-9356-c88ef5e1fbc4","Type":"ContainerStarted","Data":"bafdfdb7b687dffb572a6b55450e488064d56abc90f97048d287f1466d402ce2"}
Nov 02 14:58:20.547547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:20.547485    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n" podStartSLOduration=1.547469774 podStartE2EDuration="1.547469774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:20.546221937 +0000 UTC m=+4491.297526378" watchObservedRunningTime="2026-11-02 14:58:20.547469774 +0000 UTC m=+4491.298774212"
Nov 02 14:58:20.676324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:20.676229    5034 patch_prober.go:29] interesting pod/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:58:20.676324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:20.676295    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:58:21.394404 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:21.394322    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b39766e-2b9e-4941-bf05-9ea55c506925" path="/var/lib/kubelet/pods/5b39766e-2b9e-4941-bf05-9ea55c506925/volumes"
Nov 02 14:58:21.394909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:21.394892    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c0415b-c21f-4867-80cc-5bb953cb0532" path="/var/lib/kubelet/pods/b2c0415b-c21f-4867-80cc-5bb953cb0532/volumes"
Nov 02 14:58:21.395430 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:21.395413    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1" path="/var/lib/kubelet/pods/bbd677a1-311f-40ab-a9e8-f6c8c7c27ab1/volumes"
Nov 02 14:58:21.396224 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:21.396206    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06878a8-2080-4a80-a243-d37ec2ed7076" path="/var/lib/kubelet/pods/e06878a8-2080-4a80-a243-d37ec2ed7076/volumes"
Nov 02 14:58:21.504488 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:21.504451    5034 generic.go:356] "Generic (PLEG): container finished" podID="896ac00f-5537-4e47-9356-c88ef5e1fbc4" containerID="b97fd39238fda2936b81cb853cec8f074a33a33e3abdd1bdd4785092e89de194" exitCode=0
Nov 02 14:58:21.504883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:21.504532    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n" event={"ID":"896ac00f-5537-4e47-9356-c88ef5e1fbc4","Type":"ContainerDied","Data":"b97fd39238fda2936b81cb853cec8f074a33a33e3abdd1bdd4785092e89de194"}
Nov 02 14:58:22.669063 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.669036    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:22.791816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.791725    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-xks-ca\") pod \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") "
Nov 02 14:58:22.791816 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.791769    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896ac00f-5537-4e47-9356-c88ef5e1fbc4-kube-api-access-rnd6m\" (UniqueName: \"kubernetes.io/projected/896ac00f-5537-4e47-9356-c88ef5e1fbc4-kube-api-access-rnd6m\") pod \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") "
Nov 02 14:58:22.792056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.791938    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-openshift-ca\") pod \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\" (UID: \"896ac00f-5537-4e47-9356-c88ef5e1fbc4\") "
Nov 02 14:58:22.792357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.792324    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-openshift-ca" pod "896ac00f-5537-4e47-9356-c88ef5e1fbc4" (UID: "896ac00f-5537-4e47-9356-c88ef5e1fbc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:22.793965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.793925    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-xks-ca" pod "896ac00f-5537-4e47-9356-c88ef5e1fbc4" (UID: "896ac00f-5537-4e47-9356-c88ef5e1fbc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:22.793965 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.793949    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896ac00f-5537-4e47-9356-c88ef5e1fbc4-kube-api-access-rnd6m" pod "896ac00f-5537-4e47-9356-c88ef5e1fbc4" (UID: "896ac00f-5537-4e47-9356-c88ef5e1fbc4"). InnerVolumeSpecName "kube-api-access-rnd6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:22.893547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.893514    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:22.893547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.893546    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896ac00f-5537-4e47-9356-c88ef5e1fbc4-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:22.893547 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:22.893556    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnd6m\" (UniqueName: \"kubernetes.io/projected/896ac00f-5537-4e47-9356-c88ef5e1fbc4-kube-api-access-rnd6m\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.040981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.040936    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj" podUID="04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:23.040981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.040960    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv" podUID="39520cdc-cdca-4c22-81ef-480183b2d652" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:23.520545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.520513    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"
Nov 02 14:58:23.520667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.520521    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n" event={"ID":"896ac00f-5537-4e47-9356-c88ef5e1fbc4","Type":"ContainerDied","Data":"bafdfdb7b687dffb572a6b55450e488064d56abc90f97048d287f1466d402ce2"}
Nov 02 14:58:23.520667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.520642    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bafdfdb7b687dffb572a6b55450e488064d56abc90f97048d287f1466d402ce2"
Nov 02 14:58:23.781417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.781391    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:58:23.905364 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905335    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istiod-ca-cert\") pod \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") "
Nov 02 14:58:23.905554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905402    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-credential-socket\") pod \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") "
Nov 02 14:58:23.905554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905428    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-certs\") pod \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") "
Nov 02 14:58:23.905554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905449    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-envoy\") pod \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") "
Nov 02 14:58:23.905554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905476    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-socket\") pod \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") "
Nov 02 14:58:23.905554 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905497    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-token\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-token\") pod \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") "
Nov 02 14:58:23.905889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905574    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-kube-api-access-lw6gs\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-kube-api-access-lw6gs\") pod \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") "
Nov 02 14:58:23.905889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905652    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-data\") pod \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") "
Nov 02 14:58:23.905889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905811    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-certs" pod "8f0928e6-229e-4d23-82b7-04042ffdc0c4" (UID: "8f0928e6-229e-4d23-82b7-04042ffdc0c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.905889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905834    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-credential-socket" pod "8f0928e6-229e-4d23-82b7-04042ffdc0c4" (UID: "8f0928e6-229e-4d23-82b7-04042ffdc0c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.905889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905821    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-podinfo\") pod \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\" (UID: \"8f0928e6-229e-4d23-82b7-04042ffdc0c4\") "
Nov 02 14:58:23.905889 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.905856    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istiod-ca-cert" pod "8f0928e6-229e-4d23-82b7-04042ffdc0c4" (UID: "8f0928e6-229e-4d23-82b7-04042ffdc0c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:23.906211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.906171    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-socket" pod "8f0928e6-229e-4d23-82b7-04042ffdc0c4" (UID: "8f0928e6-229e-4d23-82b7-04042ffdc0c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.906296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.906273    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-data" pod "8f0928e6-229e-4d23-82b7-04042ffdc0c4" (UID: "8f0928e6-229e-4d23-82b7-04042ffdc0c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.906385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.906363    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.906460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.906390    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.906460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.906404    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.906460 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.906417    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:23.908459 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.908431    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-envoy" pod "8f0928e6-229e-4d23-82b7-04042ffdc0c4" (UID: "8f0928e6-229e-4d23-82b7-04042ffdc0c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:23.908975 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.908941    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-token" pod "8f0928e6-229e-4d23-82b7-04042ffdc0c4" (UID: "8f0928e6-229e-4d23-82b7-04042ffdc0c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:23.909219 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.909188    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-kube-api-access-lw6gs" pod "8f0928e6-229e-4d23-82b7-04042ffdc0c4" (UID: "8f0928e6-229e-4d23-82b7-04042ffdc0c4"). InnerVolumeSpecName "kube-api-access-lw6gs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:23.909362 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:23.909343    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-podinfo" pod "8f0928e6-229e-4d23-82b7-04042ffdc0c4" (UID: "8f0928e6-229e-4d23-82b7-04042ffdc0c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:58:24.007428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.007391    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw6gs\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-kube-api-access-lw6gs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:24.007428 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.007431    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:24.007664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.007450    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:24.007664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.007464    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:24.007664 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.007480    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0928e6-229e-4d23-82b7-04042ffdc0c4-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:24.154190 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.154151    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"]
Nov 02 14:58:24.540017 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.539979    5034 generic.go:356] "Generic (PLEG): container finished" podID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerID="3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77" exitCode=0
Nov 02 14:58:24.540471 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.540443    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" event={"ID":"8f0928e6-229e-4d23-82b7-04042ffdc0c4","Type":"ContainerDied","Data":"3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77"}
Nov 02 14:58:24.540870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.540843    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv" event={"ID":"8f0928e6-229e-4d23-82b7-04042ffdc0c4","Type":"ContainerDied","Data":"d3c6f9d2f89443f1484ae2ab5ac4bb0cc2d8c97918757c1805d287ce778b71d4"}
Nov 02 14:58:24.541041 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.541028    5034 scope.go:122] "RemoveContainer" containerID="3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77"
Nov 02 14:58:24.541177 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.540768    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"
Nov 02 14:58:24.587092 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.587070    5034 scope.go:122] "RemoveContainer" containerID="3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77"
Nov 02 14:58:24.598806 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:24.591090    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77\": container with ID starting with 3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77 not found: ID does not exist" containerID="3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77"
Nov 02 14:58:24.598806 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.591130    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77"} err="failed to get container status \"3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77\": rpc error: code = NotFound desc = could not find container \"3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77\": container with ID starting with 3a3a2a6c995a0dd14abde6a4effa29d03f386241155facea97528926b5ac4c77 not found: ID does not exist"
Nov 02 14:58:24.650393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.650349    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"]
Nov 02 14:58:24.650590 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:24.650415    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fcefcc-openshift-default-7966bbc96f-w8tcv"]
Nov 02 14:58:25.396118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:25.396082    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" path="/var/lib/kubelet/pods/8f0928e6-229e-4d23-82b7-04042ffdc0c4/volumes"
Nov 02 14:58:26.038709 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:26.038667    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"]
Nov 02 14:58:26.051475 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:26.051438    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a38947-mrvnq"]
Nov 02 14:58:27.394201 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:27.394172    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb73b587-e811-412c-aa3a-7c16ded971db" path="/var/lib/kubelet/pods/cb73b587-e811-412c-aa3a-7c16ded971db/volumes"
Nov 02 14:58:28.458038 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:28.458001    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr" podUID="ecd91416-7106-4f87-80d6-3d283e12c1a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:28.458473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:28.458331    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m" podUID="89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:28.990655 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:28.990620    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q" podUID="e28fcc5d-ad6f-426e-a370-5f6576bf37d6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:30.003979 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.003943    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw" podUID="2358bb25-5f5b-42a4-9678-852be7940e34" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 14:58:30.920497 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.920464    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"]
Nov 02 14:58:30.921019 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.921002    5034 cpu_manager.go:414] "containerMap: removing container" podUID="896ac00f-5537-4e47-9356-c88ef5e1fbc4" containerName="revoke-keys"
Nov 02 14:58:30.921080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.921020    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896ac00f-5537-4e47-9356-c88ef5e1fbc4" containerName="revoke-keys"
Nov 02 14:58:30.921080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.921046    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy"
Nov 02 14:58:30.921080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.921052    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy"
Nov 02 14:58:30.921189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.921140    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="896ac00f-5537-4e47-9356-c88ef5e1fbc4" containerName="revoke-keys"
Nov 02 14:58:30.921189 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.921154    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0928e6-229e-4d23-82b7-04042ffdc0c4" containerName="istio-proxy"
Nov 02 14:58:30.926236 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.926214    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:30.943542 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.943514    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"]
Nov 02 14:58:30.952894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.952868    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:58:30.953125 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.953096    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:58:30.954894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.954874    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:58:30.955152 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:30.955134    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:58:31.020865 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.020837    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93cadfad-92ggx\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.021276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.020890    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfchn\" (UniqueName: \"kubernetes.io/projected/6c3756a5-7cbc-4241-b939-36bcc63204de-kube-api-access-xfchn\") pod \"maas-api-revoke-sub-93cadfad-92ggx\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.021276 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.020908    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93cadfad-92ggx\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.042303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.042276    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:58:31.052569 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.052547    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 14:58:31.122074 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.122045    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93cadfad-92ggx\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.122183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.122089    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfchn\" (UniqueName: \"kubernetes.io/projected/6c3756a5-7cbc-4241-b939-36bcc63204de-kube-api-access-xfchn\") pod \"maas-api-revoke-sub-93cadfad-92ggx\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.122183 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.122109    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93cadfad-92ggx\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.122729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.122706    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93cadfad-92ggx\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.124746 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.124723    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93cadfad-92ggx\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.146470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.146444    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfchn\" (UniqueName: \"kubernetes.io/projected/6c3756a5-7cbc-4241-b939-36bcc63204de-kube-api-access-xfchn\") pod \"maas-api-revoke-sub-93cadfad-92ggx\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.240147 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.240115    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:31.544576 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:58:31.544547    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c3756a5_7cbc_4241_b939_36bcc63204de.slice/crio-30cc1ed6dba2b9a5fbd509d5e45afb9f1a71352e2c4f5a33d3b5bba8f7a4e0be WatchSource:0}: Error finding container 30cc1ed6dba2b9a5fbd509d5e45afb9f1a71352e2c4f5a33d3b5bba8f7a4e0be: Status 404 returned error can't find the container with id 30cc1ed6dba2b9a5fbd509d5e45afb9f1a71352e2c4f5a33d3b5bba8f7a4e0be
Nov 02 14:58:31.545035 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.545009    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"]
Nov 02 14:58:31.589454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.589426    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx" event={"ID":"6c3756a5-7cbc-4241-b939-36bcc63204de","Type":"ContainerStarted","Data":"30cc1ed6dba2b9a5fbd509d5e45afb9f1a71352e2c4f5a33d3b5bba8f7a4e0be"}
Nov 02 14:58:31.644322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.644291    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"]
Nov 02 14:58:31.678638 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:31.678614    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"]
Nov 02 14:58:32.601049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.601017    5034 generic.go:356] "Generic (PLEG): container finished" podID="6c3756a5-7cbc-4241-b939-36bcc63204de" containerID="06ef77ce75f03c7d215b694ecfb7ced64d7e81b6abf917b07911e621afa6be77" exitCode=0
Nov 02 14:58:32.601484 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.601096    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx" event={"ID":"6c3756a5-7cbc-4241-b939-36bcc63204de","Type":"ContainerDied","Data":"06ef77ce75f03c7d215b694ecfb7ced64d7e81b6abf917b07911e621afa6be77"}
Nov 02 14:58:32.602716 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.602691    5034 generic.go:356] "Generic (PLEG): container finished" podID="39520cdc-cdca-4c22-81ef-480183b2d652" containerID="58c9471215efcafc5c11fa233163667d1cc42e84a0d68993aa12dd3311f624e8" exitCode=0
Nov 02 14:58:32.602828 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.602720    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv" event={"ID":"39520cdc-cdca-4c22-81ef-480183b2d652","Type":"ContainerDied","Data":"58c9471215efcafc5c11fa233163667d1cc42e84a0d68993aa12dd3311f624e8"}
Nov 02 14:58:32.764886 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.764862    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 14:58:32.793367 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.793344    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 14:58:32.941835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.941745    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-plugins-config-volume\") pod \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\" (UID: \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\") "
Nov 02 14:58:32.941835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.941811    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39520cdc-cdca-4c22-81ef-480183b2d652-kube-api-access-jjnm9\" (UniqueName: \"kubernetes.io/projected/39520cdc-cdca-4c22-81ef-480183b2d652-kube-api-access-jjnm9\") pod \"39520cdc-cdca-4c22-81ef-480183b2d652\" (UID: \"39520cdc-cdca-4c22-81ef-480183b2d652\") "
Nov 02 14:58:32.942120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.941926    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-kube-api-access-sb7fw\" (UniqueName: \"kubernetes.io/projected/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-kube-api-access-sb7fw\") pod \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\" (UID: \"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8\") "
Nov 02 14:58:32.942120 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.942033    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39520cdc-cdca-4c22-81ef-480183b2d652-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39520cdc-cdca-4c22-81ef-480183b2d652-plugins-config-volume\") pod \"39520cdc-cdca-4c22-81ef-480183b2d652\" (UID: \"39520cdc-cdca-4c22-81ef-480183b2d652\") "
Nov 02 14:58:32.942243 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.942184    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-plugins-config-volume" pod "04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" (UID: "04e58a1f-e4b4-44f6-9b75-0fb12c933ff8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:32.942457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.942430    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39520cdc-cdca-4c22-81ef-480183b2d652-plugins-config-volume" pod "39520cdc-cdca-4c22-81ef-480183b2d652" (UID: "39520cdc-cdca-4c22-81ef-480183b2d652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:32.942457 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.942446    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:32.944052 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.944029    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-kube-api-access-sb7fw" pod "04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" (UID: "04e58a1f-e4b4-44f6-9b75-0fb12c933ff8"). InnerVolumeSpecName "kube-api-access-sb7fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:32.944153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:32.944074    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39520cdc-cdca-4c22-81ef-480183b2d652-kube-api-access-jjnm9" pod "39520cdc-cdca-4c22-81ef-480183b2d652" (UID: "39520cdc-cdca-4c22-81ef-480183b2d652"). InnerVolumeSpecName "kube-api-access-jjnm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:33.043972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.043943    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb7fw\" (UniqueName: \"kubernetes.io/projected/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8-kube-api-access-sb7fw\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:33.043972 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.043968    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39520cdc-cdca-4c22-81ef-480183b2d652-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:33.044169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.043980    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnm9\" (UniqueName: \"kubernetes.io/projected/39520cdc-cdca-4c22-81ef-480183b2d652-kube-api-access-jjnm9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:33.612669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.612635    5034 generic.go:356] "Generic (PLEG): container finished" podID="04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" containerID="a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380" exitCode=0
Nov 02 14:58:33.613130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.612681    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"
Nov 02 14:58:33.613130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.612719    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj" event={"ID":"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8","Type":"ContainerDied","Data":"a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380"}
Nov 02 14:58:33.613130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.612811    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj" event={"ID":"04e58a1f-e4b4-44f6-9b75-0fb12c933ff8","Type":"ContainerDied","Data":"2fecb54a81f3057afbbafd75dc964af6e27a298e916ff49ee809d46032dcae6b"}
Nov 02 14:58:33.613130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.612847    5034 scope.go:122] "RemoveContainer" containerID="a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380"
Nov 02 14:58:33.614381 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.614359    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"
Nov 02 14:58:33.614527 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.614457    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv" event={"ID":"39520cdc-cdca-4c22-81ef-480183b2d652","Type":"ContainerDied","Data":"2ace25600c64b2fd76b2f246037d959362a8310c63bc82408eb84104fed2e64f"}
Nov 02 14:58:33.623890 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.623864    5034 scope.go:122] "RemoveContainer" containerID="a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380"
Nov 02 14:58:33.624176 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:33.624151    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380\": container with ID starting with a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380 not found: ID does not exist" containerID="a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380"
Nov 02 14:58:33.624247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.624179    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380"} err="failed to get container status \"a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380\": rpc error: code = NotFound desc = could not find container \"a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380\": container with ID starting with a750c8db5c12ab02aaffdde18f7c42f14e0c129f49ef6b9f34092f116463f380 not found: ID does not exist"
Nov 02 14:58:33.624247 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.624196    5034 scope.go:122] "RemoveContainer" containerID="58c9471215efcafc5c11fa233163667d1cc42e84a0d68993aa12dd3311f624e8"
Nov 02 14:58:33.684169 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.684137    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"]
Nov 02 14:58:33.698485 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.698455    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-92f249-749647d98c-5cqnj"]
Nov 02 14:58:33.734277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.734246    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"]
Nov 02 14:58:33.739566 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.739535    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-92f249-6dc695df6f-5k2vv"]
Nov 02 14:58:33.769854 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.769833    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:33.852049 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.852021    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-openshift-ca\") pod \"6c3756a5-7cbc-4241-b939-36bcc63204de\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") "
Nov 02 14:58:33.852240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.852063    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c3756a5-7cbc-4241-b939-36bcc63204de-kube-api-access-xfchn\" (UniqueName: \"kubernetes.io/projected/6c3756a5-7cbc-4241-b939-36bcc63204de-kube-api-access-xfchn\") pod \"6c3756a5-7cbc-4241-b939-36bcc63204de\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") "
Nov 02 14:58:33.852240 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.852127    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-xks-ca\") pod \"6c3756a5-7cbc-4241-b939-36bcc63204de\" (UID: \"6c3756a5-7cbc-4241-b939-36bcc63204de\") "
Nov 02 14:58:33.852447 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.852424    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-openshift-ca" pod "6c3756a5-7cbc-4241-b939-36bcc63204de" (UID: "6c3756a5-7cbc-4241-b939-36bcc63204de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:33.854232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.854202    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-xks-ca" pod "6c3756a5-7cbc-4241-b939-36bcc63204de" (UID: "6c3756a5-7cbc-4241-b939-36bcc63204de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:33.854358 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.854331    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c3756a5-7cbc-4241-b939-36bcc63204de-kube-api-access-xfchn" pod "6c3756a5-7cbc-4241-b939-36bcc63204de" (UID: "6c3756a5-7cbc-4241-b939-36bcc63204de"). InnerVolumeSpecName "kube-api-access-xfchn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:33.953012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.952941    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:33.953012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.952970    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c3756a5-7cbc-4241-b939-36bcc63204de-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:33.953012 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:33.952985    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfchn\" (UniqueName: \"kubernetes.io/projected/6c3756a5-7cbc-4241-b939-36bcc63204de-kube-api-access-xfchn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:34.622881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:34.622847    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"
Nov 02 14:58:34.622881 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:34.622860    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx" event={"ID":"6c3756a5-7cbc-4241-b939-36bcc63204de","Type":"ContainerDied","Data":"30cc1ed6dba2b9a5fbd509d5e45afb9f1a71352e2c4f5a33d3b5bba8f7a4e0be"}
Nov 02 14:58:34.623389 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:34.622906    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cc1ed6dba2b9a5fbd509d5e45afb9f1a71352e2c4f5a33d3b5bba8f7a4e0be"
Nov 02 14:58:35.250962 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.250935    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:58:35.265071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.265050    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd91416-7106-4f87-80d6-3d283e12c1a5-kube-api-access-xfwrn\" (UniqueName: \"kubernetes.io/projected/ecd91416-7106-4f87-80d6-3d283e12c1a5-kube-api-access-xfwrn\") pod \"ecd91416-7106-4f87-80d6-3d283e12c1a5\" (UID: \"ecd91416-7106-4f87-80d6-3d283e12c1a5\") "
Nov 02 14:58:35.265195 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.265086    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd91416-7106-4f87-80d6-3d283e12c1a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd91416-7106-4f87-80d6-3d283e12c1a5-plugins-config-volume\") pod \"ecd91416-7106-4f87-80d6-3d283e12c1a5\" (UID: \"ecd91416-7106-4f87-80d6-3d283e12c1a5\") "
Nov 02 14:58:35.265465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.265445    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd91416-7106-4f87-80d6-3d283e12c1a5-plugins-config-volume" pod "ecd91416-7106-4f87-80d6-3d283e12c1a5" (UID: "ecd91416-7106-4f87-80d6-3d283e12c1a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:35.267324 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.267299    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd91416-7106-4f87-80d6-3d283e12c1a5-kube-api-access-xfwrn" pod "ecd91416-7106-4f87-80d6-3d283e12c1a5" (UID: "ecd91416-7106-4f87-80d6-3d283e12c1a5"). InnerVolumeSpecName "kube-api-access-xfwrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:35.299453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.299431    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:58:35.366109 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.366079    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-kube-api-access-n2r8p\" (UniqueName: \"kubernetes.io/projected/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-kube-api-access-n2r8p\") pod \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\" (UID: \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\") "
Nov 02 14:58:35.366237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.366172    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-plugins-config-volume\") pod \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\" (UID: \"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a\") "
Nov 02 14:58:35.366553 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.366528    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfwrn\" (UniqueName: \"kubernetes.io/projected/ecd91416-7106-4f87-80d6-3d283e12c1a5-kube-api-access-xfwrn\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:35.366669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.366560    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd91416-7106-4f87-80d6-3d283e12c1a5-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:35.366669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.366525    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-plugins-config-volume" pod "89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" (UID: "89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:35.368305 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.368281    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-kube-api-access-n2r8p" pod "89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" (UID: "89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a"). InnerVolumeSpecName "kube-api-access-n2r8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:35.393868 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.393843    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" path="/var/lib/kubelet/pods/04e58a1f-e4b4-44f6-9b75-0fb12c933ff8/volumes"
Nov 02 14:58:35.394363 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.394346    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39520cdc-cdca-4c22-81ef-480183b2d652" path="/var/lib/kubelet/pods/39520cdc-cdca-4c22-81ef-480183b2d652/volumes"
Nov 02 14:58:35.467696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.467673    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2r8p\" (UniqueName: \"kubernetes.io/projected/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-kube-api-access-n2r8p\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:35.467696 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.467699    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:35.632802 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.632740    5034 generic.go:356] "Generic (PLEG): container finished" podID="89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" containerID="ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67" exitCode=0
Nov 02 14:58:35.633278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.632808    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"
Nov 02 14:58:35.633278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.632822    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m" event={"ID":"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a","Type":"ContainerDied","Data":"ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67"}
Nov 02 14:58:35.633278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.632879    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m" event={"ID":"89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a","Type":"ContainerDied","Data":"6a5cfe28f0bfb5ddf55ed0221ff7e5fdbe2b8b8fd4f545423c8c0e1582f3f001"}
Nov 02 14:58:35.633278 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.632904    5034 scope.go:122] "RemoveContainer" containerID="ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67"
Nov 02 14:58:35.634755 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.634730    5034 generic.go:356] "Generic (PLEG): container finished" podID="ecd91416-7106-4f87-80d6-3d283e12c1a5" containerID="7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158" exitCode=0
Nov 02 14:58:35.634894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.634798    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr" event={"ID":"ecd91416-7106-4f87-80d6-3d283e12c1a5","Type":"ContainerDied","Data":"7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158"}
Nov 02 14:58:35.634894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.634845    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr" event={"ID":"ecd91416-7106-4f87-80d6-3d283e12c1a5","Type":"ContainerDied","Data":"b436a21c277fd59de50ff085a3afae02870a45779a48edb2be1f39fe6e6dd579"}
Nov 02 14:58:35.634894 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.634802    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"
Nov 02 14:58:35.643738 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.643722    5034 scope.go:122] "RemoveContainer" containerID="ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67"
Nov 02 14:58:35.644022 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:35.644003    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67\": container with ID starting with ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67 not found: ID does not exist" containerID="ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67"
Nov 02 14:58:35.644080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.644028    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67"} err="failed to get container status \"ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67\": rpc error: code = NotFound desc = could not find container \"ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67\": container with ID starting with ce5918e4601867b010f95bee370777ba9ee9fd642af5a71f9cf92a2aedd66a67 not found: ID does not exist"
Nov 02 14:58:35.644080 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.644045    5034 scope.go:122] "RemoveContainer" containerID="7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158"
Nov 02 14:58:35.653155 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.653139    5034 scope.go:122] "RemoveContainer" containerID="7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158"
Nov 02 14:58:35.653401 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:35.653385    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158\": container with ID starting with 7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158 not found: ID does not exist" containerID="7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158"
Nov 02 14:58:35.653455 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.653406    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158"} err="failed to get container status \"7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158\": rpc error: code = NotFound desc = could not find container \"7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158\": container with ID starting with 7b3066282173a19d661862dfb0b0f647502e041f8bada5a1f41b6e735c572158 not found: ID does not exist"
Nov 02 14:58:35.687574 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.687542    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"]
Nov 02 14:58:35.700342 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.700318    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fcefcc-5cc866584c-hwf6m"]
Nov 02 14:58:35.735981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.735957    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"]
Nov 02 14:58:35.739898 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:35.739876    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fcefcc-664db6b56c-wb8mr"]
Nov 02 14:58:36.035600 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.035567    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"]
Nov 02 14:58:36.040087 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.040063    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4e0b26e-s4xnr"]
Nov 02 14:58:36.537804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.535825    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 14:58:36.577273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.577241    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 14:58:36.579399 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.579367    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2358bb25-5f5b-42a4-9678-852be7940e34-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2358bb25-5f5b-42a4-9678-852be7940e34-plugins-config-volume\") pod \"2358bb25-5f5b-42a4-9678-852be7940e34\" (UID: \"2358bb25-5f5b-42a4-9678-852be7940e34\") "
Nov 02 14:58:36.579564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.579493    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2358bb25-5f5b-42a4-9678-852be7940e34-kube-api-access-4498b\" (UniqueName: \"kubernetes.io/projected/2358bb25-5f5b-42a4-9678-852be7940e34-kube-api-access-4498b\") pod \"2358bb25-5f5b-42a4-9678-852be7940e34\" (UID: \"2358bb25-5f5b-42a4-9678-852be7940e34\") "
Nov 02 14:58:36.580179 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.580152    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2358bb25-5f5b-42a4-9678-852be7940e34-plugins-config-volume" pod "2358bb25-5f5b-42a4-9678-852be7940e34" (UID: "2358bb25-5f5b-42a4-9678-852be7940e34"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:36.582285 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.582264    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2358bb25-5f5b-42a4-9678-852be7940e34-kube-api-access-4498b" pod "2358bb25-5f5b-42a4-9678-852be7940e34" (UID: "2358bb25-5f5b-42a4-9678-852be7940e34"). InnerVolumeSpecName "kube-api-access-4498b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:36.644757 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.644711    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm" podUID="721fe7a3-b75c-4796-9efb-6dac45991d50" containerName="payload-processing" containerID="cri-o://7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d" gracePeriod=25
Nov 02 14:58:36.649079 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.649049    5034 generic.go:356] "Generic (PLEG): container finished" podID="2358bb25-5f5b-42a4-9678-852be7940e34" containerID="bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685" exitCode=0
Nov 02 14:58:36.649235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.649145    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw" event={"ID":"2358bb25-5f5b-42a4-9678-852be7940e34","Type":"ContainerDied","Data":"bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685"}
Nov 02 14:58:36.649235 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.649220    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw" event={"ID":"2358bb25-5f5b-42a4-9678-852be7940e34","Type":"ContainerDied","Data":"f11e2b36adba1219225e1bdaeeb6b54a019d7d2c357642c2ce1158713641a8c4"}
Nov 02 14:58:36.649354 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.649251    5034 scope.go:122] "RemoveContainer" containerID="bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685"
Nov 02 14:58:36.649415 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.649382    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"
Nov 02 14:58:36.652057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.652027    5034 generic.go:356] "Generic (PLEG): container finished" podID="e28fcc5d-ad6f-426e-a370-5f6576bf37d6" containerID="117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab" exitCode=0
Nov 02 14:58:36.652181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.652162    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q" event={"ID":"e28fcc5d-ad6f-426e-a370-5f6576bf37d6","Type":"ContainerDied","Data":"117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab"}
Nov 02 14:58:36.652254 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.652213    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q" event={"ID":"e28fcc5d-ad6f-426e-a370-5f6576bf37d6","Type":"ContainerDied","Data":"10b71409d184378746825d4c7cc6c7917f32c09d391fa28e222c8c9aa48642e5"}
Nov 02 14:58:36.652311 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.652292    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"
Nov 02 14:58:36.667394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.667371    5034 scope.go:122] "RemoveContainer" containerID="bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685"
Nov 02 14:58:36.667714 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:36.667689    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685\": container with ID starting with bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685 not found: ID does not exist" containerID="bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685"
Nov 02 14:58:36.667836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.667722    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685"} err="failed to get container status \"bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685\": rpc error: code = NotFound desc = could not find container \"bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685\": container with ID starting with bd9172a6347268adf7acebd51ebb39a8f88c19dc7ebe4a593d7c838ae08cd685 not found: ID does not exist"
Nov 02 14:58:36.667836 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.667743    5034 scope.go:122] "RemoveContainer" containerID="117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab"
Nov 02 14:58:36.679111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.679084    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp" podUID="31dbfc8e-31ad-4f65-9520-da48fb31cc62" containerName="payload-pre-processing" containerID="cri-o://b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725" gracePeriod=25
Nov 02 14:58:36.680663 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.680633    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-kube-api-access-5nh7j\" (UniqueName: \"kubernetes.io/projected/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-kube-api-access-5nh7j\") pod \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\" (UID: \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\") "
Nov 02 14:58:36.680821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.680770    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-plugins-config-volume\") pod \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\" (UID: \"e28fcc5d-ad6f-426e-a370-5f6576bf37d6\") "
Nov 02 14:58:36.681222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.681196    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-plugins-config-volume" pod "e28fcc5d-ad6f-426e-a370-5f6576bf37d6" (UID: "e28fcc5d-ad6f-426e-a370-5f6576bf37d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:36.681322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.681210    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4498b\" (UniqueName: \"kubernetes.io/projected/2358bb25-5f5b-42a4-9678-852be7940e34-kube-api-access-4498b\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:36.681322 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.681257    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2358bb25-5f5b-42a4-9678-852be7940e34-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:36.682736 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.682714    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-kube-api-access-5nh7j" pod "e28fcc5d-ad6f-426e-a370-5f6576bf37d6" (UID: "e28fcc5d-ad6f-426e-a370-5f6576bf37d6"). InnerVolumeSpecName "kube-api-access-5nh7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:36.780945 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.780908    5034 scope.go:122] "RemoveContainer" containerID="117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab"
Nov 02 14:58:36.781412 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:36.781373    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab\": container with ID starting with 117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab not found: ID does not exist" containerID="117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab"
Nov 02 14:58:36.781544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.781426    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab"} err="failed to get container status \"117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab\": rpc error: code = NotFound desc = could not find container \"117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab\": container with ID starting with 117a920b3f8eaf373c3ed09485a415a05f4212c96d1d8c418a324b6937a294ab not found: ID does not exist"
Nov 02 14:58:36.782277 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.782247    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nh7j\" (UniqueName: \"kubernetes.io/projected/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-kube-api-access-5nh7j\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:36.782395 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.782280    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e28fcc5d-ad6f-426e-a370-5f6576bf37d6-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:36.832388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.827238    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"]
Nov 02 14:58:36.832388 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:36.830731    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-ad42e0-5dc4f77db7-ktmxw"]
Nov 02 14:58:37.055032 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.054985    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"]
Nov 02 14:58:37.059809 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.059759    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-ad42e0-6ffdd54d7-f9b7q"]
Nov 02 14:58:37.103839 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.103195    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 14:58:37.163198 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.163168    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 14:58:37.192981 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.192950    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dbfc8e-31ad-4f65-9520-da48fb31cc62-kube-api-access-82hxt\" (UniqueName: \"kubernetes.io/projected/31dbfc8e-31ad-4f65-9520-da48fb31cc62-kube-api-access-82hxt\") pod \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\" (UID: \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\") "
Nov 02 14:58:37.193135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.193008    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31dbfc8e-31ad-4f65-9520-da48fb31cc62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dbfc8e-31ad-4f65-9520-da48fb31cc62-plugins-config-volume\") pod \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\" (UID: \"31dbfc8e-31ad-4f65-9520-da48fb31cc62\") "
Nov 02 14:58:37.193135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.193077    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721fe7a3-b75c-4796-9efb-6dac45991d50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721fe7a3-b75c-4796-9efb-6dac45991d50-plugins-config-volume\") pod \"721fe7a3-b75c-4796-9efb-6dac45991d50\" (UID: \"721fe7a3-b75c-4796-9efb-6dac45991d50\") "
Nov 02 14:58:37.193135 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.193112    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721fe7a3-b75c-4796-9efb-6dac45991d50-kube-api-access-cbzbg\" (UniqueName: \"kubernetes.io/projected/721fe7a3-b75c-4796-9efb-6dac45991d50-kube-api-access-cbzbg\") pod \"721fe7a3-b75c-4796-9efb-6dac45991d50\" (UID: \"721fe7a3-b75c-4796-9efb-6dac45991d50\") "
Nov 02 14:58:37.193528 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.193496    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dbfc8e-31ad-4f65-9520-da48fb31cc62-plugins-config-volume" pod "31dbfc8e-31ad-4f65-9520-da48fb31cc62" (UID: "31dbfc8e-31ad-4f65-9520-da48fb31cc62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:37.193677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.193522    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721fe7a3-b75c-4796-9efb-6dac45991d50-plugins-config-volume" pod "721fe7a3-b75c-4796-9efb-6dac45991d50" (UID: "721fe7a3-b75c-4796-9efb-6dac45991d50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:37.196013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.195985    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dbfc8e-31ad-4f65-9520-da48fb31cc62-kube-api-access-82hxt" pod "31dbfc8e-31ad-4f65-9520-da48fb31cc62" (UID: "31dbfc8e-31ad-4f65-9520-da48fb31cc62"). InnerVolumeSpecName "kube-api-access-82hxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:37.196013 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.196009    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721fe7a3-b75c-4796-9efb-6dac45991d50-kube-api-access-cbzbg" pod "721fe7a3-b75c-4796-9efb-6dac45991d50" (UID: "721fe7a3-b75c-4796-9efb-6dac45991d50"). InnerVolumeSpecName "kube-api-access-cbzbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:37.294060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.293979    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hxt\" (UniqueName: \"kubernetes.io/projected/31dbfc8e-31ad-4f65-9520-da48fb31cc62-kube-api-access-82hxt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:37.294060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.294012    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dbfc8e-31ad-4f65-9520-da48fb31cc62-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:37.294060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.294021    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/721fe7a3-b75c-4796-9efb-6dac45991d50-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:37.294060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.294031    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbzbg\" (UniqueName: \"kubernetes.io/projected/721fe7a3-b75c-4796-9efb-6dac45991d50-kube-api-access-cbzbg\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:37.395641 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.395605    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2358bb25-5f5b-42a4-9678-852be7940e34" path="/var/lib/kubelet/pods/2358bb25-5f5b-42a4-9678-852be7940e34/volumes"
Nov 02 14:58:37.396056 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.396040    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" path="/var/lib/kubelet/pods/89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a/volumes"
Nov 02 14:58:37.396417 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.396405    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32a0860-4a22-4b17-830e-7493e26abcb3" path="/var/lib/kubelet/pods/d32a0860-4a22-4b17-830e-7493e26abcb3/volumes"
Nov 02 14:58:37.397095 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.397078    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28fcc5d-ad6f-426e-a370-5f6576bf37d6" path="/var/lib/kubelet/pods/e28fcc5d-ad6f-426e-a370-5f6576bf37d6/volumes"
Nov 02 14:58:37.397465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.397452    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd91416-7106-4f87-80d6-3d283e12c1a5" path="/var/lib/kubelet/pods/ecd91416-7106-4f87-80d6-3d283e12c1a5/volumes"
Nov 02 14:58:37.661817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.661693    5034 generic.go:356] "Generic (PLEG): container finished" podID="721fe7a3-b75c-4796-9efb-6dac45991d50" containerID="7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d" exitCode=0
Nov 02 14:58:37.661817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.661741    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm" event={"ID":"721fe7a3-b75c-4796-9efb-6dac45991d50","Type":"ContainerDied","Data":"7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d"}
Nov 02 14:58:37.661817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.661799    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"
Nov 02 14:58:37.661817 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.661811    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm" event={"ID":"721fe7a3-b75c-4796-9efb-6dac45991d50","Type":"ContainerDied","Data":"d18d5919c27b6297e9b5a58b7b9401c73caddaf289bcd423929ceb47592d8d60"}
Nov 02 14:58:37.662435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.661828    5034 scope.go:122] "RemoveContainer" containerID="7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d"
Nov 02 14:58:37.666055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.663710    5034 generic.go:356] "Generic (PLEG): container finished" podID="31dbfc8e-31ad-4f65-9520-da48fb31cc62" containerID="b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725" exitCode=0
Nov 02 14:58:37.666055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.663993    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"
Nov 02 14:58:37.666055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.664463    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp" event={"ID":"31dbfc8e-31ad-4f65-9520-da48fb31cc62","Type":"ContainerDied","Data":"b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725"}
Nov 02 14:58:37.666055 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.664505    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp" event={"ID":"31dbfc8e-31ad-4f65-9520-da48fb31cc62","Type":"ContainerDied","Data":"12e5385d11aaf9d6f65cfb4bf6236de48861e85e57b7c39e80089d1684be036d"}
Nov 02 14:58:37.676165 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.676140    5034 scope.go:122] "RemoveContainer" containerID="7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d"
Nov 02 14:58:37.676540 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:37.676507    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d\": container with ID starting with 7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d not found: ID does not exist" containerID="7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d"
Nov 02 14:58:37.676656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.676544    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d"} err="failed to get container status \"7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d\": rpc error: code = NotFound desc = could not find container \"7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d\": container with ID starting with 7fe9ee99c45033b2801396db24dc4fc9347b6ea960b1f202a8d475643be64b8d not found: ID does not exist"
Nov 02 14:58:37.676656 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.676569    5034 scope.go:122] "RemoveContainer" containerID="b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725"
Nov 02 14:58:37.695127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.695045    5034 scope.go:122] "RemoveContainer" containerID="b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725"
Nov 02 14:58:37.697585 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:37.695357    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725\": container with ID starting with b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725 not found: ID does not exist" containerID="b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725"
Nov 02 14:58:37.697585 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.695394    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725"} err="failed to get container status \"b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725\": rpc error: code = NotFound desc = could not find container \"b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725\": container with ID starting with b08524938e19e5a01fb1581b59ca8d911fb02b9eb6fd11b5aa2f3b5f14e21725 not found: ID does not exist"
Nov 02 14:58:37.730833 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.728148    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"]
Nov 02 14:58:37.735827 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.732337    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-0969b1-6dd9b48557-6ldbm"]
Nov 02 14:58:37.763491 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.763465    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"]
Nov 02 14:58:37.770180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:37.770155    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-0969b1-6df69d8d6c-54msp"]
Nov 02 14:58:39.175877 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.175837    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main" containerID="cri-o://9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c" gracePeriod=45
Nov 02 14:58:39.395290 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.395255    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31dbfc8e-31ad-4f65-9520-da48fb31cc62" path="/var/lib/kubelet/pods/31dbfc8e-31ad-4f65-9520-da48fb31cc62/volumes"
Nov 02 14:58:39.395729 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.395696    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721fe7a3-b75c-4796-9efb-6dac45991d50" path="/var/lib/kubelet/pods/721fe7a3-b75c-4796-9efb-6dac45991d50/volumes"
Nov 02 14:58:39.450555 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.450532    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:58:39.525524 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.525482    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-kube-api-access-zcvtb\" (UniqueName: \"kubernetes.io/projected/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-kube-api-access-zcvtb\") pod \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") "
Nov 02 14:58:39.525721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.525577    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-model-cache\") pod \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") "
Nov 02 14:58:39.525721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.525660    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-dshm\") pod \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") "
Nov 02 14:58:39.525721 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.525719    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tmp-dir\") pod \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") "
Nov 02 14:58:39.526118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.525766    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-home\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-home\") pod \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") "
Nov 02 14:58:39.526118 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.525827    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tls-certs\" (UniqueName: \"kubernetes.io/secret/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tls-certs\") pod \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\" (UID: \"8d1d8c55-9c6e-435b-8f68-36b3fbedb914\") "
Nov 02 14:58:39.527371 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.527337    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-home" pod "8d1d8c55-9c6e-435b-8f68-36b3fbedb914" (UID: "8d1d8c55-9c6e-435b-8f68-36b3fbedb914"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:39.527499 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.527414    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tmp-dir" pod "8d1d8c55-9c6e-435b-8f68-36b3fbedb914" (UID: "8d1d8c55-9c6e-435b-8f68-36b3fbedb914"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:39.527920 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.527894    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-model-cache" pod "8d1d8c55-9c6e-435b-8f68-36b3fbedb914" (UID: "8d1d8c55-9c6e-435b-8f68-36b3fbedb914"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:39.528487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.528455    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tls-certs" pod "8d1d8c55-9c6e-435b-8f68-36b3fbedb914" (UID: "8d1d8c55-9c6e-435b-8f68-36b3fbedb914"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:58:39.529967 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.529940    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-kube-api-access-zcvtb" pod "8d1d8c55-9c6e-435b-8f68-36b3fbedb914" (UID: "8d1d8c55-9c6e-435b-8f68-36b3fbedb914"). InnerVolumeSpecName "kube-api-access-zcvtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:39.530217 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.530192    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-dshm" pod "8d1d8c55-9c6e-435b-8f68-36b3fbedb914" (UID: "8d1d8c55-9c6e-435b-8f68-36b3fbedb914"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:39.626857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.626824    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.626857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.626853    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.626857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.626862    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.627111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.626871    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcvtb\" (UniqueName: \"kubernetes.io/projected/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-kube-api-access-zcvtb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.627111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.626880    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.627111 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.626888    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d1d8c55-9c6e-435b-8f68-36b3fbedb914-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:39.688536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.688461    5034 generic.go:356] "Generic (PLEG): container finished" podID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerID="9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c" exitCode=0
Nov 02 14:58:39.688536 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.688510    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"
Nov 02 14:58:39.688717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.688548    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" event={"ID":"8d1d8c55-9c6e-435b-8f68-36b3fbedb914","Type":"ContainerDied","Data":"9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c"}
Nov 02 14:58:39.688717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.688601    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb" event={"ID":"8d1d8c55-9c6e-435b-8f68-36b3fbedb914","Type":"ContainerDied","Data":"1b1ea9831a630a1cd4618d8d58120c93e85bb952049adad46c7b7eaa12144222"}
Nov 02 14:58:39.688717 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.688625    5034 scope.go:122] "RemoveContainer" containerID="9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c"
Nov 02 14:58:39.699301 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.699276    5034 scope.go:122] "RemoveContainer" containerID="9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c"
Nov 02 14:58:39.699582 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:39.699560    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c\": container with ID starting with 9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c not found: ID does not exist" containerID="9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c"
Nov 02 14:58:39.699654 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.699589    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c"} err="failed to get container status \"9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c\": rpc error: code = NotFound desc = could not find container \"9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c\": container with ID starting with 9b335fc28261ed3bb67862e04c1e990fa438bdd2cb88a5d9ff20b58348080c0c not found: ID does not exist"
Nov 02 14:58:39.749502 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.749477    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"]
Nov 02 14:58:39.753470 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:39.753447    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-0969b1/simulator-w2-0969b1-kserve-698b79d64d-s5chb"]
Nov 02 14:58:41.394313 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:41.394277    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" path="/var/lib/kubelet/pods/8d1d8c55-9c6e-435b-8f68-36b3fbedb914/volumes"
Nov 02 14:58:44.926379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:44.926346    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"]
Nov 02 14:58:44.926903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:44.926635    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy" containerID="cri-o://ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1" gracePeriod=30
Nov 02 14:58:45.031943 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:45.031912    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"]
Nov 02 14:58:45.035882 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:45.035850    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01232c80-sxkh6"]
Nov 02 14:58:45.392504 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:45.392470    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a363cd-a08f-4ffe-8272-a6b7943407a7" path="/var/lib/kubelet/pods/a8a363cd-a08f-4ffe-8272-a6b7943407a7/volumes"
Nov 02 14:58:48.414354 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:48.414315    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:58:50.201494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.201469    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:58:50.220127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220067    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53cf1eb-35ca-4451-816a-f352391a736b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53cf1eb-35ca-4451-816a-f352391a736b-istiod-ca-cert\") pod \"c53cf1eb-35ca-4451-816a-f352391a736b\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") "
Nov 02 14:58:50.220127 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220106    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-kube-api-access-kz4nd\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-kube-api-access-kz4nd\") pod \"c53cf1eb-35ca-4451-816a-f352391a736b\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") "
Nov 02 14:58:50.220338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220142    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-socket\") pod \"c53cf1eb-35ca-4451-816a-f352391a736b\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") "
Nov 02 14:58:50.220338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220261    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-certs\") pod \"c53cf1eb-35ca-4451-816a-f352391a736b\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") "
Nov 02 14:58:50.220338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220324    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-credential-socket\") pod \"c53cf1eb-35ca-4451-816a-f352391a736b\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") "
Nov 02 14:58:50.220507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220403    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c53cf1eb-35ca-4451-816a-f352391a736b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53cf1eb-35ca-4451-816a-f352391a736b-istio-podinfo\") pod \"c53cf1eb-35ca-4451-816a-f352391a736b\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") "
Nov 02 14:58:50.220507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220464    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-envoy\") pod \"c53cf1eb-35ca-4451-816a-f352391a736b\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") "
Nov 02 14:58:50.220507 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220446    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-socket" pod "c53cf1eb-35ca-4451-816a-f352391a736b" (UID: "c53cf1eb-35ca-4451-816a-f352391a736b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.220672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220526    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-istio-token\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-istio-token\") pod \"c53cf1eb-35ca-4451-816a-f352391a736b\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") "
Nov 02 14:58:50.220672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220575    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-data\") pod \"c53cf1eb-35ca-4451-816a-f352391a736b\" (UID: \"c53cf1eb-35ca-4451-816a-f352391a736b\") "
Nov 02 14:58:50.220672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220530    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53cf1eb-35ca-4451-816a-f352391a736b-istiod-ca-cert" pod "c53cf1eb-35ca-4451-816a-f352391a736b" (UID: "c53cf1eb-35ca-4451-816a-f352391a736b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:58:50.220672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220606    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-certs" pod "c53cf1eb-35ca-4451-816a-f352391a736b" (UID: "c53cf1eb-35ca-4451-816a-f352391a736b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.220672 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220640    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-credential-socket" pod "c53cf1eb-35ca-4451-816a-f352391a736b" (UID: "c53cf1eb-35ca-4451-816a-f352391a736b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.220952 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220887    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-data" pod "c53cf1eb-35ca-4451-816a-f352391a736b" (UID: "c53cf1eb-35ca-4451-816a-f352391a736b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.221024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.220995    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.221024 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.221013    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53cf1eb-35ca-4451-816a-f352391a736b-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.221131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.221027    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.221131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.221041    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.221131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.221056    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.222573 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.222544    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-kube-api-access-kz4nd" pod "c53cf1eb-35ca-4451-816a-f352391a736b" (UID: "c53cf1eb-35ca-4451-816a-f352391a736b"). InnerVolumeSpecName "kube-api-access-kz4nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:50.222795 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.222757    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c53cf1eb-35ca-4451-816a-f352391a736b-istio-podinfo" pod "c53cf1eb-35ca-4451-816a-f352391a736b" (UID: "c53cf1eb-35ca-4451-816a-f352391a736b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:58:50.222896 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.222810    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-envoy" pod "c53cf1eb-35ca-4451-816a-f352391a736b" (UID: "c53cf1eb-35ca-4451-816a-f352391a736b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:58:50.222990 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.222969    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-istio-token" pod "c53cf1eb-35ca-4451-816a-f352391a736b" (UID: "c53cf1eb-35ca-4451-816a-f352391a736b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:58:50.322465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.322426    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53cf1eb-35ca-4451-816a-f352391a736b-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.322465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.322460    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53cf1eb-35ca-4451-816a-f352391a736b-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.322465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.322469    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.322708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.322498    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz4nd\" (UniqueName: \"kubernetes.io/projected/c53cf1eb-35ca-4451-816a-f352391a736b-kube-api-access-kz4nd\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:58:50.768426 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.768388    5034 generic.go:356] "Generic (PLEG): container finished" podID="c53cf1eb-35ca-4451-816a-f352391a736b" containerID="ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1" exitCode=0
Nov 02 14:58:50.768631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.768453    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"
Nov 02 14:58:50.768631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.768484    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" event={"ID":"c53cf1eb-35ca-4451-816a-f352391a736b","Type":"ContainerDied","Data":"ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1"}
Nov 02 14:58:50.768631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.768557    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4" event={"ID":"c53cf1eb-35ca-4451-816a-f352391a736b","Type":"ContainerDied","Data":"e5d75581d67a0396e7b65787b3ee9bc526b67f6f1ddb443d6bf268f8191d5d91"}
Nov 02 14:58:50.768631 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.768589    5034 scope.go:122] "RemoveContainer" containerID="ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1"
Nov 02 14:58:50.779915 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.779899    5034 scope.go:122] "RemoveContainer" containerID="ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1"
Nov 02 14:58:50.780195 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:58:50.780175    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1\": container with ID starting with ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1 not found: ID does not exist" containerID="ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1"
Nov 02 14:58:50.780293 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.780206    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1"} err="failed to get container status \"ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1\": rpc error: code = NotFound desc = could not find container \"ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1\": container with ID starting with ba0d2cb324e62ecc2b612100bf9f038f7ed006a6c59034bf59c07092daad88f1 not found: ID does not exist"
Nov 02 14:58:50.822808 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.822771    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"]
Nov 02 14:58:50.836338 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:50.836313    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-ad42e0-openshift-default-5dcc558457-s4kh4"]
Nov 02 14:58:51.393275 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:51.393241    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" path="/var/lib/kubelet/pods/c53cf1eb-35ca-4451-816a-f352391a736b/volumes"
Nov 02 14:58:59.134982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:59.134942    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"]
Nov 02 14:58:59.135552 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:58:59.135213    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" containerID="cri-o://d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959" gracePeriod=30
Nov 02 14:59:00.119269 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:00.119230    5034 patch_prober.go:29] interesting pod/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 14:59:00.119450 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:00.119299    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 14:59:00.711211 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:00.711180    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67755d48d8-2qsfp_0a8b5d62-dc9c-4383-b54c-b26d25258d29/authorino/0.log"
Nov 02 14:59:00.994453 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:00.994427    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cscl8_fc17acb2-e694-4585-8c2b-3fed98a4df14/manager/0.log"
Nov 02 14:59:01.035651 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:01.035622    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"]
Nov 02 14:59:01.048357 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:01.048328    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-894a26f8-7svbf"]
Nov 02 14:59:01.393596 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:01.393512    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8042d30f-134f-46bd-819c-218563ea6cc6" path="/var/lib/kubelet/pods/8042d30f-134f-46bd-819c-218563ea6cc6/volumes"
Nov 02 14:59:01.719961 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:01.719881    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bwz2h_58393844-0472-4a85-ba31-9586f62ea2c2/manager/0.log"
Nov 02 14:59:02.028431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:02.028396    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"]
Nov 02 14:59:02.040857 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:02.040824    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afaa709a-qrh6v"]
Nov 02 14:59:03.392977 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:03.392937    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586" path="/var/lib/kubelet/pods/c0b6c22d-86cd-4f3f-bc9a-0bbdd0da8586/volumes"
Nov 02 14:59:04.392924 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.392899    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 14:59:04.453760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.453696    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-socket\") pod \"fd355c18-4dfd-45e3-848f-7d462d728762\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") "
Nov 02 14:59:04.453760 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.453744    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-kube-api-access-cgmxx\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-kube-api-access-cgmxx\") pod \"fd355c18-4dfd-45e3-848f-7d462d728762\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") "
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.453787    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd355c18-4dfd-45e3-848f-7d462d728762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd355c18-4dfd-45e3-848f-7d462d728762-istio-podinfo\") pod \"fd355c18-4dfd-45e3-848f-7d462d728762\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") "
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.453826    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd355c18-4dfd-45e3-848f-7d462d728762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd355c18-4dfd-45e3-848f-7d462d728762-istiod-ca-cert\") pod \"fd355c18-4dfd-45e3-848f-7d462d728762\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") "
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.453863    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-envoy\") pod \"fd355c18-4dfd-45e3-848f-7d462d728762\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") "
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.453907    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-certs\") pod \"fd355c18-4dfd-45e3-848f-7d462d728762\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") "
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.453970    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-istio-token\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-istio-token\") pod \"fd355c18-4dfd-45e3-848f-7d462d728762\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") "
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.453999    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-socket" pod "fd355c18-4dfd-45e3-848f-7d462d728762" (UID: "fd355c18-4dfd-45e3-848f-7d462d728762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.454031    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-credential-socket\") pod \"fd355c18-4dfd-45e3-848f-7d462d728762\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") "
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.454079    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-data\") pod \"fd355c18-4dfd-45e3-848f-7d462d728762\" (UID: \"fd355c18-4dfd-45e3-848f-7d462d728762\") "
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.454181    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-certs" pod "fd355c18-4dfd-45e3-848f-7d462d728762" (UID: "fd355c18-4dfd-45e3-848f-7d462d728762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.454538    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.454570    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.454537    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-credential-socket" pod "fd355c18-4dfd-45e3-848f-7d462d728762" (UID: "fd355c18-4dfd-45e3-848f-7d462d728762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:59:04.454646 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.454621    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd355c18-4dfd-45e3-848f-7d462d728762-istiod-ca-cert" pod "fd355c18-4dfd-45e3-848f-7d462d728762" (UID: "fd355c18-4dfd-45e3-848f-7d462d728762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:59:04.455463 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.454920    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-data" pod "fd355c18-4dfd-45e3-848f-7d462d728762" (UID: "fd355c18-4dfd-45e3-848f-7d462d728762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:59:04.456862 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.456836    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd355c18-4dfd-45e3-848f-7d462d728762-istio-podinfo" pod "fd355c18-4dfd-45e3-848f-7d462d728762" (UID: "fd355c18-4dfd-45e3-848f-7d462d728762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 14:59:04.457004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.456912    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-envoy" pod "fd355c18-4dfd-45e3-848f-7d462d728762" (UID: "fd355c18-4dfd-45e3-848f-7d462d728762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 14:59:04.457004 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.456976    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-kube-api-access-cgmxx" pod "fd355c18-4dfd-45e3-848f-7d462d728762" (UID: "fd355c18-4dfd-45e3-848f-7d462d728762"). InnerVolumeSpecName "kube-api-access-cgmxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:59:04.458593 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.458567    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-istio-token" pod "fd355c18-4dfd-45e3-848f-7d462d728762" (UID: "fd355c18-4dfd-45e3-848f-7d462d728762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:59:04.555805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.555744    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:04.555951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.555813    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgmxx\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-kube-api-access-cgmxx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:04.555951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.555831    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd355c18-4dfd-45e3-848f-7d462d728762-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:04.555951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.555844    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd355c18-4dfd-45e3-848f-7d462d728762-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:04.555951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.555855    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:04.555951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.555866    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd355c18-4dfd-45e3-848f-7d462d728762-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:04.555951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.555877    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd355c18-4dfd-45e3-848f-7d462d728762-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:04.866408 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.866369    5034 generic.go:356] "Generic (PLEG): container finished" podID="fd355c18-4dfd-45e3-848f-7d462d728762" containerID="d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959" exitCode=0
Nov 02 14:59:04.866564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.866414    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"
Nov 02 14:59:04.866564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.866451    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" event={"ID":"fd355c18-4dfd-45e3-848f-7d462d728762","Type":"ContainerDied","Data":"d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959"}
Nov 02 14:59:04.866564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.866513    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg" event={"ID":"fd355c18-4dfd-45e3-848f-7d462d728762","Type":"ContainerDied","Data":"6e895e70b4b8a5b86ffed36fedda1ea1e5a624036b3bfc40c28333969a215db1"}
Nov 02 14:59:04.866564 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.866538    5034 scope.go:122] "RemoveContainer" containerID="d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959"
Nov 02 14:59:04.879443 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.879221    5034 scope.go:122] "RemoveContainer" containerID="d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959"
Nov 02 14:59:04.879592 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:59:04.879567    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959\": container with ID starting with d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959 not found: ID does not exist" containerID="d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959"
Nov 02 14:59:04.879667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.879599    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959"} err="failed to get container status \"d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959\": rpc error: code = NotFound desc = could not find container \"d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959\": container with ID starting with d42f1768fedd7b4ed88f713e73d6d7ec67edcf96266c5b085cdfb469e1016959 not found: ID does not exist"
Nov 02 14:59:04.953325 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.953298    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"]
Nov 02 14:59:04.966192 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:04.966164    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-0969b1-openshift-default-6cdcfd698c-fgxmg"]
Nov 02 14:59:05.393393 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:05.393361    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" path="/var/lib/kubelet/pods/fd355c18-4dfd-45e3-848f-7d462d728762/volumes"
Nov 02 14:59:13.029544 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:13.029467    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"]
Nov 02 14:59:13.042180 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:13.042156    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e07dbdd7-2vv77"]
Nov 02 14:59:13.392845 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:13.392747    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9e0676-6e51-4517-90ee-4c945a632e57" path="/var/lib/kubelet/pods/cf9e0676-6e51-4517-90ee-4c945a632e57/volumes"
Nov 02 14:59:14.538191 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:14.538162    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gdvr6_7e043e8d-d6b7-4f76-af4d-46b513dcd3f6/server/0.log"
Nov 02 14:59:18.412230 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:59:18.412194    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:59:21.668769 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.668735    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"]
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669175    5034 cpu_manager.go:414] "containerMap: removing container" podUID="721fe7a3-b75c-4796-9efb-6dac45991d50" containerName="payload-processing"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669188    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721fe7a3-b75c-4796-9efb-6dac45991d50" containerName="payload-processing"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669203    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e28fcc5d-ad6f-426e-a370-5f6576bf37d6" containerName="payload-pre-processing"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669211    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28fcc5d-ad6f-426e-a370-5f6576bf37d6" containerName="payload-pre-processing"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669224    5034 cpu_manager.go:414] "containerMap: removing container" podUID="31dbfc8e-31ad-4f65-9520-da48fb31cc62" containerName="payload-pre-processing"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669229    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dbfc8e-31ad-4f65-9520-da48fb31cc62" containerName="payload-pre-processing"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669238    5034 cpu_manager.go:414] "containerMap: removing container" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669244    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669255    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ecd91416-7106-4f87-80d6-3d283e12c1a5" containerName="payload-processing"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669260    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd91416-7106-4f87-80d6-3d283e12c1a5" containerName="payload-processing"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669269    5034 cpu_manager.go:414] "containerMap: removing container" podUID="39520cdc-cdca-4c22-81ef-480183b2d652" containerName="payload-processing"
Nov 02 14:59:21.669272 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669274    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39520cdc-cdca-4c22-81ef-480183b2d652" containerName="payload-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669285    5034 cpu_manager.go:414] "containerMap: removing container" podUID="2358bb25-5f5b-42a4-9678-852be7940e34" containerName="payload-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669293    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2358bb25-5f5b-42a4-9678-852be7940e34" containerName="payload-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669304    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669311    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669320    5034 cpu_manager.go:414] "containerMap: removing container" podUID="04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" containerName="payload-pre-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669325    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" containerName="payload-pre-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669338    5034 cpu_manager.go:414] "containerMap: removing container" podUID="89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" containerName="payload-pre-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669343    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" containerName="payload-pre-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669355    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669362    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669373    5034 cpu_manager.go:414] "containerMap: removing container" podUID="6c3756a5-7cbc-4241-b939-36bcc63204de" containerName="revoke-keys"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669380    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c3756a5-7cbc-4241-b939-36bcc63204de" containerName="revoke-keys"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669456    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28fcc5d-ad6f-426e-a370-5f6576bf37d6" containerName="payload-pre-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669466    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="31dbfc8e-31ad-4f65-9520-da48fb31cc62" containerName="payload-pre-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669471    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd355c18-4dfd-45e3-848f-7d462d728762" containerName="istio-proxy"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669480    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd91416-7106-4f87-80d6-3d283e12c1a5" containerName="payload-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669486    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="39520cdc-cdca-4c22-81ef-480183b2d652" containerName="payload-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669493    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="2358bb25-5f5b-42a4-9678-852be7940e34" containerName="payload-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669499    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1d8c55-9c6e-435b-8f68-36b3fbedb914" containerName="main"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669504    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e58a1f-e4b4-44f6-9b75-0fb12c933ff8" containerName="payload-pre-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669511    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c3ad6b-ee98-4a76-86fe-ad4e9da28e0a" containerName="payload-pre-processing"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669517    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53cf1eb-35ca-4451-816a-f352391a736b" containerName="istio-proxy"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669525    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c3756a5-7cbc-4241-b939-36bcc63204de" containerName="revoke-keys"
Nov 02 14:59:21.669937 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.669537    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="721fe7a3-b75c-4796-9efb-6dac45991d50" containerName="payload-processing"
Nov 02 14:59:21.673153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.673133    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.689978 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.689952    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fcad4d9a-openshift-default-dockercfg-qzblk\""
Nov 02 14:59:21.706525 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.706497    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"]
Nov 02 14:59:21.809950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.809920    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v844w\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-kube-api-access-v844w\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.810117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.809967    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istiod-ca-cert\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.810117 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.810059    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-socket\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.810213 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.810179    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-data\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.810260 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.810211    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-podinfo\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.810303 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.810278    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-envoy\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.810339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.810314    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-certs\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.810339 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.810331    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-token\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.810473 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.810456    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-credential-socket\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.911866 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.911830    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-credential-socket\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.911874    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v844w\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-kube-api-access-v844w\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912009    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istiod-ca-cert\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912060 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912057    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-socket\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912104    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-data\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912145    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-podinfo\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912191    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-envoy\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912251 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912238    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-certs\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912264    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-token\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912328    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-credential-socket\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912498 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912388    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-socket\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912640 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912495    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-data\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912708 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912648    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-certs\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.912840 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.912817    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istiod-ca-cert\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.914561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.914533    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-envoy\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.914829 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.914810    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-podinfo\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.942348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.942285    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-token\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.942348 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.942316    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v844w\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-kube-api-access-v844w\") pod \"e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") " pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:21.987089 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:21.987062    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:22.304057 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:59:22.304027    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc879f023_3b9a_45dd_8dd6_f7b9fefe6e35.slice/crio-e9062e473667c0100f066833932fdba28cc6dbf2d4f42effc935d6d0e5e46af7 WatchSource:0}: Error finding container e9062e473667c0100f066833932fdba28cc6dbf2d4f42effc935d6d0e5e46af7: Status 404 returned error can't find the container with id e9062e473667c0100f066833932fdba28cc6dbf2d4f42effc935d6d0e5e46af7
Nov 02 14:59:22.304057 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:22.304046    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"]
Nov 02 14:59:22.306130 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:22.306097    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:59:22.306222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:22.306161    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:59:22.306222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:22.306188    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:59:22.967756 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:22.967722    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" event={"ID":"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35","Type":"ContainerStarted","Data":"81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6"}
Nov 02 14:59:22.968161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:22.967791    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" event={"ID":"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35","Type":"ContainerStarted","Data":"e9062e473667c0100f066833932fdba28cc6dbf2d4f42effc935d6d0e5e46af7"}
Nov 02 14:59:22.987181 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:22.987152    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:23.028238 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:23.028184    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" podStartSLOduration=2.028170309 podStartE2EDuration="2.028170309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:59:23.026589598 +0000 UTC m=+4553.777894036" watchObservedRunningTime="2026-11-02 14:59:23.028170309 +0000 UTC m=+4553.779474747"
Nov 02 14:59:23.992537 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:23.992507    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:24.979273 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:24.979239    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:24.980172 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:24.980146    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:26.445194 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.445158    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"]
Nov 02 14:59:26.449379 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.449360    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:26.471232 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.471203    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"]
Nov 02 14:59:26.480947 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.480926    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fcad4d9a\""
Nov 02 14:59:26.481050 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.481018    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fcad4d9a-dockercfg-dtl6c\""
Nov 02 14:59:26.501788 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.501756    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"]
Nov 02 14:59:26.506891 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.506873    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:26.524122 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.524097    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"]
Nov 02 14:59:26.554296 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.554272    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5sdb\" (UniqueName: \"kubernetes.io/projected/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-kube-api-access-p5sdb\") pod \"payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct\" (UID: \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\") " pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:26.554445 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.554389    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct\" (UID: \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\") " pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:26.655546 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.655516    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c70b29-f79d-49c3-baa9-010ec140081a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp\" (UID: \"24c70b29-f79d-49c3-baa9-010ec140081a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:26.655719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.655577    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5sdb\" (UniqueName: \"kubernetes.io/projected/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-kube-api-access-p5sdb\") pod \"payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct\" (UID: \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\") " pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:26.655719 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.655697    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct\" (UID: \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\") " pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:26.655879 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.655753    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwlj\" (UniqueName: \"kubernetes.io/projected/24c70b29-f79d-49c3-baa9-010ec140081a-kube-api-access-brwlj\") pod \"payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp\" (UID: \"24c70b29-f79d-49c3-baa9-010ec140081a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:26.656385 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.656366    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-plugins-config-volume\") pod \"payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct\" (UID: \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\") " pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:26.679995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.679970    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5sdb\" (UniqueName: \"kubernetes.io/projected/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-kube-api-access-p5sdb\") pod \"payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct\" (UID: \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\") " pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:26.757222 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.757200    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c70b29-f79d-49c3-baa9-010ec140081a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp\" (UID: \"24c70b29-f79d-49c3-baa9-010ec140081a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:26.757334 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.757282    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwlj\" (UniqueName: \"kubernetes.io/projected/24c70b29-f79d-49c3-baa9-010ec140081a-kube-api-access-brwlj\") pod \"payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp\" (UID: \"24c70b29-f79d-49c3-baa9-010ec140081a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:26.757897 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.757878    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c70b29-f79d-49c3-baa9-010ec140081a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp\" (UID: \"24c70b29-f79d-49c3-baa9-010ec140081a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:26.763128 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.763109    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:26.781995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.781945    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwlj\" (UniqueName: \"kubernetes.io/projected/24c70b29-f79d-49c3-baa9-010ec140081a-kube-api-access-brwlj\") pod \"payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp\" (UID: \"24c70b29-f79d-49c3-baa9-010ec140081a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:26.821043 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:26.820970    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:27.062886 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:59:27.062851    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b5dccf_d169_420a_9f77_1cf4a5f8e4c3.slice/crio-1dae0c9faec824c20b2dceb836e36a08c402869a4843170307fc75b28a83f81d WatchSource:0}: Error finding container 1dae0c9faec824c20b2dceb836e36a08c402869a4843170307fc75b28a83f81d: Status 404 returned error can't find the container with id 1dae0c9faec824c20b2dceb836e36a08c402869a4843170307fc75b28a83f81d
Nov 02 14:59:27.064062 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:27.064034    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"]
Nov 02 14:59:27.138717 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:59:27.138687    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c70b29_f79d_49c3_baa9_010ec140081a.slice/crio-90fb94370fc394cae447c51f977b24f1f45e9a008e4ef374cac0822a5843c7f5 WatchSource:0}: Error finding container 90fb94370fc394cae447c51f977b24f1f45e9a008e4ef374cac0822a5843c7f5: Status 404 returned error can't find the container with id 90fb94370fc394cae447c51f977b24f1f45e9a008e4ef374cac0822a5843c7f5
Nov 02 14:59:27.139337 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:27.139312    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"]
Nov 02 14:59:27.999009 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:27.998964    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct" event={"ID":"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3","Type":"ContainerStarted","Data":"b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4"}
Nov 02 14:59:27.999523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:27.999049    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:27.999523 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:27.999072    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct" event={"ID":"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3","Type":"ContainerStarted","Data":"1dae0c9faec824c20b2dceb836e36a08c402869a4843170307fc75b28a83f81d"}
Nov 02 14:59:28.000800 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:28.000741    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp" event={"ID":"24c70b29-f79d-49c3-baa9-010ec140081a","Type":"ContainerStarted","Data":"8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28"}
Nov 02 14:59:28.000926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:28.000806    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp" event={"ID":"24c70b29-f79d-49c3-baa9-010ec140081a","Type":"ContainerStarted","Data":"90fb94370fc394cae447c51f977b24f1f45e9a008e4ef374cac0822a5843c7f5"}
Nov 02 14:59:28.000926 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:28.000850    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:28.041667 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:28.041598    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct" podStartSLOduration=2.041579039 podStartE2EDuration="2.041579039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:59:28.041055584 +0000 UTC m=+4558.792360024" watchObservedRunningTime="2026-11-02 14:59:28.041579039 +0000 UTC m=+4558.792883478"
Nov 02 14:59:28.084939 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:28.084891    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp" podStartSLOduration=2.084878083 podStartE2EDuration="2.084878083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:59:28.082160065 +0000 UTC m=+4558.833464503" watchObservedRunningTime="2026-11-02 14:59:28.084878083 +0000 UTC m=+4558.836182549"
Nov 02 14:59:39.012086 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.012059    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:39.012454 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.012127    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:39.076129 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.076099    5034 generic.go:356] "Generic (PLEG): container finished" podID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerID="81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6" exitCode=0
Nov 02 14:59:39.076333 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.076187    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" event={"ID":"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35","Type":"ContainerDied","Data":"81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6"}
Nov 02 14:59:39.076545 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.076525    5034 scope.go:122] "RemoveContainer" containerID="81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6"
Nov 02 14:59:39.077973 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.077934    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:59:39.078071 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.078028    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:59:39.078119 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.078076    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 14:59:39.900995 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.900963    5034 scope.go:122] "RemoveContainer" containerID="b55daf415eedc633c6db7ed9758ed65aad5a83cb3d366b04dbbb5ab317156f35"
Nov 02 14:59:39.910256 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.910231    5034 scope.go:122] "RemoveContainer" containerID="14e50cc250000e7dc6cd97023bf062acf273461119e64c881b7c283a94a22d88"
Nov 02 14:59:39.918434 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.918414    5034 scope.go:122] "RemoveContainer" containerID="57ae871587bb11afa5b67ada3c437552e33cebcd01638f85b605b52ce01a1bfb"
Nov 02 14:59:39.926431 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.926416    5034 scope.go:122] "RemoveContainer" containerID="f3197027dce9208fa77e4441379580afe8cadb2d3b6be8da1debaabc5ee18ac3"
Nov 02 14:59:39.984982 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:39.984888    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:40.115761 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.115719    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" event={"ID":"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35","Type":"ContainerStarted","Data":"150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5"}
Nov 02 14:59:40.444456 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.444419    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"]
Nov 02 14:59:40.448158 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.448141    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:40.468249 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.468222    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:59:40.468366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.468250    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 14:59:40.468366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.468260    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 14:59:40.468366 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.468270    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cgbzt\""
Nov 02 14:59:40.497804 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.481310    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"]
Nov 02 14:59:40.501310 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.501275    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:40.501465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.501326    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89mtz\" (UniqueName: \"kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:40.501465 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.501421    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:40.602903 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.602872    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:40.603059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.602910    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89mtz\" (UniqueName: \"kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:40.603059 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.602939    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:40.603612 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.603590    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:40.605286 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.605261    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:40.632974 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:59:40.632950    5034 projected.go:196] Error preparing data for projected volume kube-api-access-89mtz for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 14:59:40.633067 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:59:40.633021    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz podName:f3b57d0d-f58e-4946-8fb4-7554d79923a0 nodeName:}" failed. No retries permitted until 2026-11-02 14:59:41.133003524 +0000 UTC m=+4571.884307955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-89mtz" (UniqueName: "kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz") pod "maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2" (UID: "f3b57d0d-f58e-4946-8fb4-7554d79923a0") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 14:59:40.993805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.988059    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:40.993805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.989656    5034 patch_prober.go:29] interesting pod/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 02 14:59:40.993805 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:40.989704    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 02 14:59:41.209489 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:41.209445    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89mtz\" (UniqueName: \"kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:41.225242 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:59:41.225218    5034 projected.go:196] Error preparing data for projected volume kube-api-access-89mtz for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 14:59:41.225378 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:59:41.225276    5034 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz podName:f3b57d0d-f58e-4946-8fb4-7554d79923a0 nodeName:}" failed. No retries permitted until 2026-11-02 14:59:42.225261482 +0000 UTC m=+4572.976565898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-89mtz" (UniqueName: "kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz") pod "maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2" (UID: "f3b57d0d-f58e-4946-8fb4-7554d79923a0") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 02 14:59:41.991207 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:41.991170    5034 patch_prober.go:29] interesting pod/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 02 14:59:41.991534 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:41.991503    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 02 14:59:42.320244 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:42.320162    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89mtz\" (UniqueName: \"kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:42.344494 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:42.344461    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89mtz\" (UniqueName: \"kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz\") pod \"maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:42.561685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:42.561650    5034 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2" secret="" err="secret \"maas-api-cleanup-dockercfg-cgbzt\" not found"
Nov 02 14:59:42.561883 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:42.561707    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:42.897583 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:42.897553    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"]
Nov 02 14:59:42.897906 ip-10-0-139-26 kubenswrapper[5034]: W1102 14:59:42.897869    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b57d0d_f58e_4946_8fb4_7554d79923a0.slice/crio-ff933a5c7e6cac8ca460471044c79509bf4a6a719b034ed3f39752496c7eede3 WatchSource:0}: Error finding container ff933a5c7e6cac8ca460471044c79509bf4a6a719b034ed3f39752496c7eede3: Status 404 returned error can't find the container with id ff933a5c7e6cac8ca460471044c79509bf4a6a719b034ed3f39752496c7eede3
Nov 02 14:59:42.994921 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:42.994889    5034 patch_prober.go:29] interesting pod/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 02 14:59:42.995094 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:42.994950    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 02 14:59:43.133435 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:43.133404    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2" event={"ID":"f3b57d0d-f58e-4946-8fb4-7554d79923a0","Type":"ContainerStarted","Data":"ff933a5c7e6cac8ca460471044c79509bf4a6a719b034ed3f39752496c7eede3"}
Nov 02 14:59:43.993950 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:43.993912    5034 patch_prober.go:29] interesting pod/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 02 14:59:43.994341 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:43.993978    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 02 14:59:44.140821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:44.140717    5034 generic.go:356] "Generic (PLEG): container finished" podID="f3b57d0d-f58e-4946-8fb4-7554d79923a0" containerID="2f6a662cac7fed4d75b3ee7eca7be35e7cbf47c7b4f6472bf5e0bdf22ddf70b7" exitCode=0
Nov 02 14:59:44.140821 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:44.140808    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2" event={"ID":"f3b57d0d-f58e-4946-8fb4-7554d79923a0","Type":"ContainerDied","Data":"2f6a662cac7fed4d75b3ee7eca7be35e7cbf47c7b4f6472bf5e0bdf22ddf70b7"}
Nov 02 14:59:44.987831 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:44.987763    5034 patch_prober.go:29] interesting pod/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused" start-of-body=
Nov 02 14:59:44.988126 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:44.987851    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.176:15021/healthz/ready\": dial tcp 10.132.1.176:15021: connect: connection refused"
Nov 02 14:59:45.294629 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.294605    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:45.346521 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.346483    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-openshift-ca\") pod \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") "
Nov 02 14:59:45.346647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.346551    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz\" (UniqueName: \"kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz\") pod \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") "
Nov 02 14:59:45.346647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.346614    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-xks-ca\") pod \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\" (UID: \"f3b57d0d-f58e-4946-8fb4-7554d79923a0\") "
Nov 02 14:59:45.346909 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.346889    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-openshift-ca" pod "f3b57d0d-f58e-4946-8fb4-7554d79923a0" (UID: "f3b57d0d-f58e-4946-8fb4-7554d79923a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:59:45.347006 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.346992    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:45.348750 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.348724    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-xks-ca" pod "f3b57d0d-f58e-4946-8fb4-7554d79923a0" (UID: "f3b57d0d-f58e-4946-8fb4-7554d79923a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:59:45.348870 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.348754    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz" pod "f3b57d0d-f58e-4946-8fb4-7554d79923a0" (UID: "f3b57d0d-f58e-4946-8fb4-7554d79923a0"). InnerVolumeSpecName "kube-api-access-89mtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:59:45.447420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.447397    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89mtz\" (UniqueName: \"kubernetes.io/projected/f3b57d0d-f58e-4946-8fb4-7554d79923a0-kube-api-access-89mtz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:45.447420 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.447421    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3b57d0d-f58e-4946-8fb4-7554d79923a0-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:45.991951 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.991920    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:45.992394 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.992371    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:45.993178 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:45.993159    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 14:59:46.154318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:46.154283    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2" event={"ID":"f3b57d0d-f58e-4946-8fb4-7554d79923a0","Type":"ContainerDied","Data":"ff933a5c7e6cac8ca460471044c79509bf4a6a719b034ed3f39752496c7eede3"}
Nov 02 14:59:46.154318 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:46.154314    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"
Nov 02 14:59:46.154647 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:46.154342    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff933a5c7e6cac8ca460471044c79509bf4a6a719b034ed3f39752496c7eede3"
Nov 02 14:59:48.412822 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:59:48.412767    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 14:59:50.866607 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:50.866573    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"]
Nov 02 14:59:50.886685 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:50.886649    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"]
Nov 02 14:59:53.587487 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:53.587454    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"]
Nov 02 14:59:53.600436 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:53.600407    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fcad4d9a-7k4z2"]
Nov 02 14:59:55.393911 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:55.393863    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b57d0d-f58e-4946-8fb4-7554d79923a0" path="/var/lib/kubelet/pods/f3b57d0d-f58e-4946-8fb4-7554d79923a0/volumes"
Nov 02 14:59:55.867695 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:55.867637    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct" podUID="10b5dccf-d169-420a-9f77-1cf4a5f8e4c3" containerName="payload-processing" containerID="cri-o://b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4" gracePeriod=25
Nov 02 14:59:55.887153 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:55.887122    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp" podUID="24c70b29-f79d-49c3-baa9-010ec140081a" containerName="payload-pre-processing" containerID="cri-o://8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28" gracePeriod=25
Nov 02 14:59:56.163413 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.163386    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:56.183669 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.183646    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:56.214131 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.214096    5034 generic.go:356] "Generic (PLEG): container finished" podID="10b5dccf-d169-420a-9f77-1cf4a5f8e4c3" containerID="b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4" exitCode=0
Nov 02 14:59:56.214271 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.214252    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct" event={"ID":"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3","Type":"ContainerDied","Data":"b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4"}
Nov 02 14:59:56.214319 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.214305    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct" event={"ID":"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3","Type":"ContainerDied","Data":"1dae0c9faec824c20b2dceb836e36a08c402869a4843170307fc75b28a83f81d"}
Nov 02 14:59:56.214356 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.214335    5034 scope.go:122] "RemoveContainer" containerID="b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4"
Nov 02 14:59:56.214486 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.214471    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"
Nov 02 14:59:56.216561 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.216530    5034 generic.go:356] "Generic (PLEG): container finished" podID="24c70b29-f79d-49c3-baa9-010ec140081a" containerID="8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28" exitCode=0
Nov 02 14:59:56.216677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.216561    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp" event={"ID":"24c70b29-f79d-49c3-baa9-010ec140081a","Type":"ContainerDied","Data":"8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28"}
Nov 02 14:59:56.216677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.216583    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"
Nov 02 14:59:56.216677 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.216613    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp" event={"ID":"24c70b29-f79d-49c3-baa9-010ec140081a","Type":"ContainerDied","Data":"90fb94370fc394cae447c51f977b24f1f45e9a008e4ef374cac0822a5843c7f5"}
Nov 02 14:59:56.226953 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.226929    5034 scope.go:122] "RemoveContainer" containerID="b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4"
Nov 02 14:59:56.227249 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:59:56.227223    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4\": container with ID starting with b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4 not found: ID does not exist" containerID="b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4"
Nov 02 14:59:56.227350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.227256    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4"} err="failed to get container status \"b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4\": rpc error: code = NotFound desc = could not find container \"b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4\": container with ID starting with b81dee6d43f3ebf4ec055f3e0a1e5f70178bb4625efc7ad5211fbee3481416e4 not found: ID does not exist"
Nov 02 14:59:56.227350 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.227273    5034 scope.go:122] "RemoveContainer" containerID="8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28"
Nov 02 14:59:56.237722 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.237700    5034 scope.go:122] "RemoveContainer" containerID="8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28"
Nov 02 14:59:56.237997 ip-10-0-139-26 kubenswrapper[5034]: E1102 14:59:56.237975    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28\": container with ID starting with 8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28 not found: ID does not exist" containerID="8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28"
Nov 02 14:59:56.238066 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.238003    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28"} err="failed to get container status \"8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28\": rpc error: code = NotFound desc = could not find container \"8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28\": container with ID starting with 8746672129edb9c1ade85c5f79483b6635f659cbcff680d80bf405ecbba87a28 not found: ID does not exist"
Nov 02 14:59:56.250161 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.250139    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-kube-api-access-p5sdb\" (UniqueName: \"kubernetes.io/projected/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-kube-api-access-p5sdb\") pod \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\" (UID: \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\") "
Nov 02 14:59:56.250279 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.250196    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-plugins-config-volume\") pod \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\" (UID: \"10b5dccf-d169-420a-9f77-1cf4a5f8e4c3\") "
Nov 02 14:59:56.250279 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.250247    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c70b29-f79d-49c3-baa9-010ec140081a-kube-api-access-brwlj\" (UniqueName: \"kubernetes.io/projected/24c70b29-f79d-49c3-baa9-010ec140081a-kube-api-access-brwlj\") pod \"24c70b29-f79d-49c3-baa9-010ec140081a\" (UID: \"24c70b29-f79d-49c3-baa9-010ec140081a\") "
Nov 02 14:59:56.250424 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.250293    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c70b29-f79d-49c3-baa9-010ec140081a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c70b29-f79d-49c3-baa9-010ec140081a-plugins-config-volume\") pod \"24c70b29-f79d-49c3-baa9-010ec140081a\" (UID: \"24c70b29-f79d-49c3-baa9-010ec140081a\") "
Nov 02 14:59:56.251011 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.250984    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c70b29-f79d-49c3-baa9-010ec140081a-plugins-config-volume" pod "24c70b29-f79d-49c3-baa9-010ec140081a" (UID: "24c70b29-f79d-49c3-baa9-010ec140081a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:59:56.251294 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.251272    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-plugins-config-volume" pod "10b5dccf-d169-420a-9f77-1cf4a5f8e4c3" (UID: "10b5dccf-d169-420a-9f77-1cf4a5f8e4c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:59:56.254200 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.254175    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c70b29-f79d-49c3-baa9-010ec140081a-kube-api-access-brwlj" pod "24c70b29-f79d-49c3-baa9-010ec140081a" (UID: "24c70b29-f79d-49c3-baa9-010ec140081a"). InnerVolumeSpecName "kube-api-access-brwlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:59:56.254387 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.254367    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-kube-api-access-p5sdb" pod "10b5dccf-d169-420a-9f77-1cf4a5f8e4c3" (UID: "10b5dccf-d169-420a-9f77-1cf4a5f8e4c3"). InnerVolumeSpecName "kube-api-access-p5sdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:59:56.351237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.351202    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5sdb\" (UniqueName: \"kubernetes.io/projected/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-kube-api-access-p5sdb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:56.351237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.351231    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:56.351237 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.351242    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brwlj\" (UniqueName: \"kubernetes.io/projected/24c70b29-f79d-49c3-baa9-010ec140081a-kube-api-access-brwlj\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:56.351482 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.351252    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24c70b29-f79d-49c3-baa9-010ec140081a-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 14:59:56.562707 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.562678    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"]
Nov 02 14:59:56.575931 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.575906    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fcad4d9a-56f95cd9c5-f4bct"]
Nov 02 14:59:56.607108 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.607083    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"]
Nov 02 14:59:56.612392 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.612366    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fcad4d9a-86f4c696cf-mjdqp"]
Nov 02 14:59:56.724048 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.724017    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"]
Nov 02 14:59:56.724835 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:56.724241    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy" containerID="cri-o://150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5" gracePeriod=30
Nov 02 14:59:57.392959 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:57.392928    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b5dccf-d169-420a-9f77-1cf4a5f8e4c3" path="/var/lib/kubelet/pods/10b5dccf-d169-420a-9f77-1cf4a5f8e4c3/volumes"
Nov 02 14:59:57.393316 ip-10-0-139-26 kubenswrapper[5034]: I1102 14:59:57.393304    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c70b29-f79d-49c3-baa9-010ec140081a" path="/var/lib/kubelet/pods/24c70b29-f79d-49c3-baa9-010ec140081a/volumes"
Nov 02 15:00:00.161575 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.161540    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"]
Nov 02 15:00:00.161994 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.161947    5034 cpu_manager.go:414] "containerMap: removing container" podUID="10b5dccf-d169-420a-9f77-1cf4a5f8e4c3" containerName="payload-processing"
Nov 02 15:00:00.161994 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.161959    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b5dccf-d169-420a-9f77-1cf4a5f8e4c3" containerName="payload-processing"
Nov 02 15:00:00.161994 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.161969    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f3b57d0d-f58e-4946-8fb4-7554d79923a0" containerName="revoke-keys"
Nov 02 15:00:00.161994 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.161975    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b57d0d-f58e-4946-8fb4-7554d79923a0" containerName="revoke-keys"
Nov 02 15:00:00.162142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.162004    5034 cpu_manager.go:414] "containerMap: removing container" podUID="24c70b29-f79d-49c3-baa9-010ec140081a" containerName="payload-pre-processing"
Nov 02 15:00:00.162142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.162010    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c70b29-f79d-49c3-baa9-010ec140081a" containerName="payload-pre-processing"
Nov 02 15:00:00.162142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.162096    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c70b29-f79d-49c3-baa9-010ec140081a" containerName="payload-pre-processing"
Nov 02 15:00:00.162142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.162110    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b5dccf-d169-420a-9f77-1cf4a5f8e4c3" containerName="payload-processing"
Nov 02 15:00:00.162142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.162118    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b57d0d-f58e-4946-8fb4-7554d79923a0" containerName="revoke-keys"
Nov 02 15:00:00.166890 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.166867    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:00.197265 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.197207    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:00:00.197265 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.197226    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:00:00.197265 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.197231    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:00:00.199944 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.199920    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"]
Nov 02 15:00:00.287480 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.287455    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/da19eb78-933a-4610-92f3-9ce9ea414776-kube-api-access-c77hx\") pod \"maas-api-key-cleanup-29893860-8hxgw\" (UID: \"da19eb78-933a-4610-92f3-9ce9ea414776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:00.287627 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.287486    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da19eb78-933a-4610-92f3-9ce9ea414776-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893860-8hxgw\" (UID: \"da19eb78-933a-4610-92f3-9ce9ea414776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:00.388417 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.388386    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/da19eb78-933a-4610-92f3-9ce9ea414776-kube-api-access-c77hx\") pod \"maas-api-key-cleanup-29893860-8hxgw\" (UID: \"da19eb78-933a-4610-92f3-9ce9ea414776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:00.388417 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.388418    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da19eb78-933a-4610-92f3-9ce9ea414776-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893860-8hxgw\" (UID: \"da19eb78-933a-4610-92f3-9ce9ea414776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:00.389127 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.389106    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da19eb78-933a-4610-92f3-9ce9ea414776-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893860-8hxgw\" (UID: \"da19eb78-933a-4610-92f3-9ce9ea414776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:00.413472 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.413412    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/da19eb78-933a-4610-92f3-9ce9ea414776-kube-api-access-c77hx\") pod \"maas-api-key-cleanup-29893860-8hxgw\" (UID: \"da19eb78-933a-4610-92f3-9ce9ea414776\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:00.479611 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.479579    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:00.788087 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:00:00.788053    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda19eb78_933a_4610_92f3_9ce9ea414776.slice/crio-902be887f8657673771b7c52af297dc7c587bf71433cef62d8768dc1d06cbded WatchSource:0}: Error finding container 902be887f8657673771b7c52af297dc7c587bf71433cef62d8768dc1d06cbded: Status 404 returned error can't find the container with id 902be887f8657673771b7c52af297dc7c587bf71433cef62d8768dc1d06cbded
Nov 02 15:00:00.788215 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.788113    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"]
Nov 02 15:00:00.995239 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.995204    5034 patch_prober.go:29] interesting pod/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 15:00:00.995392 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:00.995262    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 15:00:01.249253 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:01.249217    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" event={"ID":"da19eb78-933a-4610-92f3-9ce9ea414776","Type":"ContainerStarted","Data":"9720ba366be997e89e3dbd5f747797ef6e0b6f01e3d1a3ece55752845283312e"}
Nov 02 15:00:01.249797 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:01.249280    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" event={"ID":"da19eb78-933a-4610-92f3-9ce9ea414776","Type":"ContainerStarted","Data":"902be887f8657673771b7c52af297dc7c587bf71433cef62d8768dc1d06cbded"}
Nov 02 15:00:01.292164 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:01.292104    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" podStartSLOduration=1.292089895 podStartE2EDuration="1.292089895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:01.290870229 +0000 UTC m=+4592.042174669" watchObservedRunningTime="2026-11-02 15:00:01.292089895 +0000 UTC m=+4592.043394332"
Nov 02 15:00:01.988835 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:01.988808    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 15:00:02.105352 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105250    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-podinfo\") pod \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") "
Nov 02 15:00:02.105352 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105309    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istiod-ca-cert\") pod \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") "
Nov 02 15:00:02.105613 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105351    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-credential-socket\") pod \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") "
Nov 02 15:00:02.105613 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105452    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-data\") pod \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") "
Nov 02 15:00:02.105613 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105480    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-socket\") pod \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") "
Nov 02 15:00:02.105613 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105553    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-token\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-token\") pod \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") "
Nov 02 15:00:02.105613 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105603    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-envoy\") pod \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") "
Nov 02 15:00:02.105923 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105667    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-certs\") pod \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") "
Nov 02 15:00:02.105923 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105698    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-kube-api-access-v844w\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-kube-api-access-v844w\") pod \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\" (UID: \"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35\") "
Nov 02 15:00:02.105923 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.105768    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-data" pod "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" (UID: "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:02.106105 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.106085    5034 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-data\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:02.106385 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.106317    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-socket" pod "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" (UID: "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:02.106768 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.106736    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-certs" pod "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" (UID: "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:02.107087 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.107066    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istiod-ca-cert" pod "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" (UID: "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:00:02.108306 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.108105    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-podinfo" pod "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" (UID: "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 15:00:02.108561 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.108523    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-kube-api-access-v844w" pod "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" (UID: "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35"). InnerVolumeSpecName "kube-api-access-v844w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:02.108711 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.108685    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-envoy" pod "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" (UID: "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:02.108829 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.108693    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-token" pod "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" (UID: "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:02.108901 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.108835    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-credential-socket" pod "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" (UID: "c879f023-3b9a-45dd-8dd6-f7b9fefe6e35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:00:02.207593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.207549    5034 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:02.207593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.207581    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v844w\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-kube-api-access-v844w\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:02.207593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.207591    5034 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-podinfo\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:02.207593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.207600    5034 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istiod-ca-cert\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:02.207593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.207608    5034 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-credential-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:02.208096 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.207618    5034 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-workload-socket\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:02.208096 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.207627    5034 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-token\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:02.208096 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.207636    5034 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35-istio-envoy\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:02.255886 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.255844    5034 generic.go:356] "Generic (PLEG): container finished" podID="da19eb78-933a-4610-92f3-9ce9ea414776" containerID="9720ba366be997e89e3dbd5f747797ef6e0b6f01e3d1a3ece55752845283312e" exitCode=60
Nov 02 15:00:02.256315 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.255927    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" event={"ID":"da19eb78-933a-4610-92f3-9ce9ea414776","Type":"ContainerDied","Data":"9720ba366be997e89e3dbd5f747797ef6e0b6f01e3d1a3ece55752845283312e"}
Nov 02 15:00:02.256315 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.256216    5034 scope.go:122] "RemoveContainer" containerID="9720ba366be997e89e3dbd5f747797ef6e0b6f01e3d1a3ece55752845283312e"
Nov 02 15:00:02.257909 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.257888    5034 generic.go:356] "Generic (PLEG): container finished" podID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerID="150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5" exitCode=0
Nov 02 15:00:02.258010 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.257940    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"
Nov 02 15:00:02.258010 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.257977    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" event={"ID":"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35","Type":"ContainerDied","Data":"150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5"}
Nov 02 15:00:02.258134 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.258037    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6" event={"ID":"c879f023-3b9a-45dd-8dd6-f7b9fefe6e35","Type":"ContainerDied","Data":"e9062e473667c0100f066833932fdba28cc6dbf2d4f42effc935d6d0e5e46af7"}
Nov 02 15:00:02.258134 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.258071    5034 scope.go:122] "RemoveContainer" containerID="150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5"
Nov 02 15:00:02.268971 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.268946    5034 scope.go:122] "RemoveContainer" containerID="81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6"
Nov 02 15:00:02.281720 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.281702    5034 scope.go:122] "RemoveContainer" containerID="150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5"
Nov 02 15:00:02.282241 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:00:02.282213    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5\": container with ID starting with 150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5 not found: ID does not exist" containerID="150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5"
Nov 02 15:00:02.282347 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.282251    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5"} err="failed to get container status \"150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5\": rpc error: code = NotFound desc = could not find container \"150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5\": container with ID starting with 150c2eaaad11cac18472e08ffca1db537f2fca51433f89614375a477a54983d5 not found: ID does not exist"
Nov 02 15:00:02.282347 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.282270    5034 scope.go:122] "RemoveContainer" containerID="81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6"
Nov 02 15:00:02.282622 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:00:02.282601    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6\": container with ID starting with 81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6 not found: ID does not exist" containerID="81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6"
Nov 02 15:00:02.282668 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.282631    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6"} err="failed to get container status \"81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6\": rpc error: code = NotFound desc = could not find container \"81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6\": container with ID starting with 81fa1634d45933d1ce7d3a2fbb38ba155272de6dccfb9c9b2c617fe7459b59e6 not found: ID does not exist"
Nov 02 15:00:02.337190 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.337158    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"]
Nov 02 15:00:02.348104 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:02.348071    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fcad4d9a-openshift-default-79fb47fd77-hxzf6"]
Nov 02 15:00:03.272066 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:03.272014    5034 generic.go:356] "Generic (PLEG): container finished" podID="da19eb78-933a-4610-92f3-9ce9ea414776" containerID="88c3c240a5b1e70c0b62d801b7ec113b7e71e979005990864d4e0be4f9e9e25f" exitCode=60
Nov 02 15:00:03.272818 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:03.272169    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" event={"ID":"da19eb78-933a-4610-92f3-9ce9ea414776","Type":"ContainerDied","Data":"88c3c240a5b1e70c0b62d801b7ec113b7e71e979005990864d4e0be4f9e9e25f"}
Nov 02 15:00:03.272818 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:03.272267    5034 scope.go:122] "RemoveContainer" containerID="9720ba366be997e89e3dbd5f747797ef6e0b6f01e3d1a3ece55752845283312e"
Nov 02 15:00:03.272968 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:03.272844    5034 scope.go:122] "RemoveContainer" containerID="88c3c240a5b1e70c0b62d801b7ec113b7e71e979005990864d4e0be4f9e9e25f"
Nov 02 15:00:03.273462 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:00:03.273208    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893860-8hxgw_odh-ai-gateway-infra(da19eb78-933a-4610-92f3-9ce9ea414776)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" podUID="da19eb78-933a-4610-92f3-9ce9ea414776"
Nov 02 15:00:03.393193 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:03.393159    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" path="/var/lib/kubelet/pods/c879f023-3b9a-45dd-8dd6-f7b9fefe6e35/volumes"
Nov 02 15:00:04.283988 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:04.283958    5034 scope.go:122] "RemoveContainer" containerID="88c3c240a5b1e70c0b62d801b7ec113b7e71e979005990864d4e0be4f9e9e25f"
Nov 02 15:00:04.284375 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:00:04.284198    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893860-8hxgw_odh-ai-gateway-infra(da19eb78-933a-4610-92f3-9ce9ea414776)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" podUID="da19eb78-933a-4610-92f3-9ce9ea414776"
Nov 02 15:00:14.387352 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:14.387318    5034 scope.go:122] "RemoveContainer" containerID="88c3c240a5b1e70c0b62d801b7ec113b7e71e979005990864d4e0be4f9e9e25f"
Nov 02 15:00:15.348228 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:15.348147    5034 generic.go:356] "Generic (PLEG): container finished" podID="da19eb78-933a-4610-92f3-9ce9ea414776" containerID="df5ca012b373465ae890f5f86be5321a648a3fc7b8e86d17ce63ed575e84166f" exitCode=60
Nov 02 15:00:15.348430 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:15.348220    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" event={"ID":"da19eb78-933a-4610-92f3-9ce9ea414776","Type":"ContainerDied","Data":"df5ca012b373465ae890f5f86be5321a648a3fc7b8e86d17ce63ed575e84166f"}
Nov 02 15:00:15.348430 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:15.348285    5034 scope.go:122] "RemoveContainer" containerID="88c3c240a5b1e70c0b62d801b7ec113b7e71e979005990864d4e0be4f9e9e25f"
Nov 02 15:00:15.348643 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:15.348622    5034 scope.go:122] "RemoveContainer" containerID="df5ca012b373465ae890f5f86be5321a648a3fc7b8e86d17ce63ed575e84166f"
Nov 02 15:00:15.348924 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:00:15.348904    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893860-8hxgw_odh-ai-gateway-infra(da19eb78-933a-4610-92f3-9ce9ea414776)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" podUID="da19eb78-933a-4610-92f3-9ce9ea414776"
Nov 02 15:00:16.400277 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:16.400234    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"]
Nov 02 15:00:16.547893 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:16.547864    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:16.650358 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:16.650271    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da19eb78-933a-4610-92f3-9ce9ea414776-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da19eb78-933a-4610-92f3-9ce9ea414776-maas-api-service-ca\") pod \"da19eb78-933a-4610-92f3-9ce9ea414776\" (UID: \"da19eb78-933a-4610-92f3-9ce9ea414776\") "
Nov 02 15:00:16.650358 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:16.650311    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da19eb78-933a-4610-92f3-9ce9ea414776-kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/da19eb78-933a-4610-92f3-9ce9ea414776-kube-api-access-c77hx\") pod \"da19eb78-933a-4610-92f3-9ce9ea414776\" (UID: \"da19eb78-933a-4610-92f3-9ce9ea414776\") "
Nov 02 15:00:16.650719 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:16.650681    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da19eb78-933a-4610-92f3-9ce9ea414776-maas-api-service-ca" pod "da19eb78-933a-4610-92f3-9ce9ea414776" (UID: "da19eb78-933a-4610-92f3-9ce9ea414776"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:00:16.652496 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:16.652474    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da19eb78-933a-4610-92f3-9ce9ea414776-kube-api-access-c77hx" pod "da19eb78-933a-4610-92f3-9ce9ea414776" (UID: "da19eb78-933a-4610-92f3-9ce9ea414776"). InnerVolumeSpecName "kube-api-access-c77hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:16.751219 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:16.751185    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da19eb78-933a-4610-92f3-9ce9ea414776-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:16.751219 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:16.751215    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/da19eb78-933a-4610-92f3-9ce9ea414776-kube-api-access-c77hx\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:17.362179 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:17.362145    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"
Nov 02 15:00:17.362382 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:17.362176    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw" event={"ID":"da19eb78-933a-4610-92f3-9ce9ea414776","Type":"ContainerDied","Data":"902be887f8657673771b7c52af297dc7c587bf71433cef62d8768dc1d06cbded"}
Nov 02 15:00:17.362382 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:17.362259    5034 scope.go:122] "RemoveContainer" containerID="df5ca012b373465ae890f5f86be5321a648a3fc7b8e86d17ce63ed575e84166f"
Nov 02 15:00:17.416559 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:17.416526    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"]
Nov 02 15:00:17.429390 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:17.429360    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893860-8hxgw"]
Nov 02 15:00:18.417915 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:00:18.417870    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:00:19.393268 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:19.393239    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" path="/var/lib/kubelet/pods/da19eb78-933a-4610-92f3-9ce9ea414776/volumes"
Nov 02 15:00:21.633370 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633334    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"]
Nov 02 15:00:21.633794 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633728    5034 cpu_manager.go:414] "containerMap: removing container" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" containerName="cleanup"
Nov 02 15:00:21.633794 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633739    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" containerName="cleanup"
Nov 02 15:00:21.633794 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633750    5034 cpu_manager.go:414] "containerMap: removing container" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" containerName="cleanup"
Nov 02 15:00:21.633794 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633757    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" containerName="cleanup"
Nov 02 15:00:21.633794 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633769    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy"
Nov 02 15:00:21.633794 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633793    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy"
Nov 02 15:00:21.634012 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633820    5034 cpu_manager.go:414] "containerMap: removing container" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" containerName="cleanup"
Nov 02 15:00:21.634012 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633824    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" containerName="cleanup"
Nov 02 15:00:21.634012 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633910    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy"
Nov 02 15:00:21.634012 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633922    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" containerName="cleanup"
Nov 02 15:00:21.634012 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633928    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" containerName="cleanup"
Nov 02 15:00:21.634012 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.633934    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy"
Nov 02 15:00:21.638493 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.638470    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.656793 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.656749    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"]
Nov 02 15:00:21.667414 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.667384    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:00:21.667414 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.667384    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:00:21.667414 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.667395    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:00:21.667661 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.667398    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:00:21.799220 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.799185    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkt6\" (UniqueName: \"kubernetes.io/projected/c34051f1-43e5-4b16-82e6-60d98e539607-kube-api-access-vdkt6\") pod \"maas-api-revoke-sub-85237e45-z2pmm\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.799393 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.799236    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-z2pmm\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.799393 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.799264    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-z2pmm\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.900363 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.900254    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkt6\" (UniqueName: \"kubernetes.io/projected/c34051f1-43e5-4b16-82e6-60d98e539607-kube-api-access-vdkt6\") pod \"maas-api-revoke-sub-85237e45-z2pmm\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.900363 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.900308    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-z2pmm\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.900636 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.900440    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-z2pmm\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.901156 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.901129    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-z2pmm\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.902727 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.902705    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-z2pmm\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.925170 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.925137    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkt6\" (UniqueName: \"kubernetes.io/projected/c34051f1-43e5-4b16-82e6-60d98e539607-kube-api-access-vdkt6\") pod \"maas-api-revoke-sub-85237e45-z2pmm\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:21.951436 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:21.951401    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:22.257227 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:22.257193    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"]
Nov 02 15:00:22.257693 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:00:22.257667    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34051f1_43e5_4b16_82e6_60d98e539607.slice/crio-6e6f617080162c29a02fa5726e2ae3aff1d22919db8650ef0a91bba12755be64 WatchSource:0}: Error finding container 6e6f617080162c29a02fa5726e2ae3aff1d22919db8650ef0a91bba12755be64: Status 404 returned error can't find the container with id 6e6f617080162c29a02fa5726e2ae3aff1d22919db8650ef0a91bba12755be64
Nov 02 15:00:22.393355 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:22.393320    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm" event={"ID":"c34051f1-43e5-4b16-82e6-60d98e539607","Type":"ContainerStarted","Data":"6e6f617080162c29a02fa5726e2ae3aff1d22919db8650ef0a91bba12755be64"}
Nov 02 15:00:23.401222 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:23.401183    5034 generic.go:356] "Generic (PLEG): container finished" podID="c34051f1-43e5-4b16-82e6-60d98e539607" containerID="1829b69dcef0f61fb20aa350c19d9f3d6eba703895927a8fb2a3f08b9cb97469" exitCode=0
Nov 02 15:00:23.401617 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:23.401241    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm" event={"ID":"c34051f1-43e5-4b16-82e6-60d98e539607","Type":"ContainerDied","Data":"1829b69dcef0f61fb20aa350c19d9f3d6eba703895927a8fb2a3f08b9cb97469"}
Nov 02 15:00:24.559241 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.559213    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:24.728318 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.728231    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34051f1-43e5-4b16-82e6-60d98e539607-kube-api-access-vdkt6\" (UniqueName: \"kubernetes.io/projected/c34051f1-43e5-4b16-82e6-60d98e539607-kube-api-access-vdkt6\") pod \"c34051f1-43e5-4b16-82e6-60d98e539607\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") "
Nov 02 15:00:24.728502 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.728370    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-xks-ca\") pod \"c34051f1-43e5-4b16-82e6-60d98e539607\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") "
Nov 02 15:00:24.728502 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.728396    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-openshift-ca\") pod \"c34051f1-43e5-4b16-82e6-60d98e539607\" (UID: \"c34051f1-43e5-4b16-82e6-60d98e539607\") "
Nov 02 15:00:24.728905 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.728876    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-openshift-ca" pod "c34051f1-43e5-4b16-82e6-60d98e539607" (UID: "c34051f1-43e5-4b16-82e6-60d98e539607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:00:24.730487 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.730455    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-xks-ca" pod "c34051f1-43e5-4b16-82e6-60d98e539607" (UID: "c34051f1-43e5-4b16-82e6-60d98e539607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:00:24.730582 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.730483    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34051f1-43e5-4b16-82e6-60d98e539607-kube-api-access-vdkt6" pod "c34051f1-43e5-4b16-82e6-60d98e539607" (UID: "c34051f1-43e5-4b16-82e6-60d98e539607"). InnerVolumeSpecName "kube-api-access-vdkt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:00:24.829114 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.829076    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:24.829114 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.829106    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c34051f1-43e5-4b16-82e6-60d98e539607-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:24.829114 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:24.829115    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdkt6\" (UniqueName: \"kubernetes.io/projected/c34051f1-43e5-4b16-82e6-60d98e539607-kube-api-access-vdkt6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:00:25.414157 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:25.414124    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"
Nov 02 15:00:25.414157 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:25.414149    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm" event={"ID":"c34051f1-43e5-4b16-82e6-60d98e539607","Type":"ContainerDied","Data":"6e6f617080162c29a02fa5726e2ae3aff1d22919db8650ef0a91bba12755be64"}
Nov 02 15:00:25.414439 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:25.414197    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e6f617080162c29a02fa5726e2ae3aff1d22919db8650ef0a91bba12755be64"
Nov 02 15:00:27.523073 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.523028    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"]
Nov 02 15:00:27.523624 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.523559    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c34051f1-43e5-4b16-82e6-60d98e539607" containerName="revoke-keys"
Nov 02 15:00:27.523624 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.523584    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34051f1-43e5-4b16-82e6-60d98e539607" containerName="revoke-keys"
Nov 02 15:00:27.523624 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.523612    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy"
Nov 02 15:00:27.523624 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.523618    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c879f023-3b9a-45dd-8dd6-f7b9fefe6e35" containerName="istio-proxy"
Nov 02 15:00:27.523845 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.523716    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34051f1-43e5-4b16-82e6-60d98e539607" containerName="revoke-keys"
Nov 02 15:00:27.523845 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.523730    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="da19eb78-933a-4610-92f3-9ce9ea414776" containerName="cleanup"
Nov 02 15:00:27.527336 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.527315    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.540273 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.540244    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"]
Nov 02 15:00:27.543819 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.543752    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"]
Nov 02 15:00:27.543951 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.543865    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.547511 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.547486    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.550855 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.550767    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-27gvt\""
Nov 02 15:00:27.552296 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552272    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.552412 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552309    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.552412 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552335    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.552412 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552360    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.552412 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552389    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.552663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552414    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.552663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552448    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.552663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552478    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzlhg\" (UniqueName: \"kubernetes.io/projected/6b68886e-53d7-470d-8d43-893f733dae8f-kube-api-access-fzlhg\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.552663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552508    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qxz\" (UniqueName: \"kubernetes.io/projected/dcf337cc-18f5-4451-88fe-351cf905dca8-kube-api-access-95qxz\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.552663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552534    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.552663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552565    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.552663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552593    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf337cc-18f5-4451-88fe-351cf905dca8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.552663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552620    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.552663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552654    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43892ef8-c4cc-414e-bc41-d1626001a0fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552708    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552751    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552801    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552835    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552879    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b68886e-53d7-470d-8d43-893f733dae8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552913    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552937    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b68886e-53d7-470d-8d43-893f733dae8f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552968    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.552998    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.553025    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.553077    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.553103    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4djd\" (UniqueName: \"kubernetes.io/projected/43892ef8-c4cc-414e-bc41-d1626001a0fa-kube-api-access-m4djd\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.553361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.553130    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b68886e-53d7-470d-8d43-893f733dae8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.561471 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.561442    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"]
Nov 02 15:00:27.567602 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.566499    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"]
Nov 02 15:00:27.578893 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.574443    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.578893 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.574739    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"]
Nov 02 15:00:27.592055 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.592024    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"]
Nov 02 15:00:27.595904 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.595873    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"]
Nov 02 15:00:27.654279 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654241    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzlhg\" (UniqueName: \"kubernetes.io/projected/6b68886e-53d7-470d-8d43-893f733dae8f-kube-api-access-fzlhg\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.654491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654290    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qxz\" (UniqueName: \"kubernetes.io/projected/dcf337cc-18f5-4451-88fe-351cf905dca8-kube-api-access-95qxz\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.654491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654318    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.654491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654347    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.654491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654373    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf337cc-18f5-4451-88fe-351cf905dca8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.654491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654400    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.654491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654458    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565989d2-f1a9-4413-a50d-fd54da149761-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654507    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43892ef8-c4cc-414e-bc41-d1626001a0fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654546    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654594    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654647    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654681    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654716    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654742    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565989d2-f1a9-4413-a50d-fd54da149761-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654796    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b68886e-53d7-470d-8d43-893f733dae8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654827    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654851    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b68886e-53d7-470d-8d43-893f733dae8f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654864    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654867    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654880    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.654897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654883    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.654952    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655011    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655048    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655081    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655113    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565989d2-f1a9-4413-a50d-fd54da149761-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655142    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655310    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655338    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf337cc-18f5-4451-88fe-351cf905dca8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655352    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43892ef8-c4cc-414e-bc41-d1626001a0fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655593    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655603    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4djd\" (UniqueName: \"kubernetes.io/projected/43892ef8-c4cc-414e-bc41-d1626001a0fa-kube-api-access-m4djd\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655643    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dg9d\" (UniqueName: \"kubernetes.io/projected/565989d2-f1a9-4413-a50d-fd54da149761-kube-api-access-9dg9d\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.655695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655673    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b68886e-53d7-470d-8d43-893f733dae8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655712    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655741    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655767    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655812    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655843    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655737    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655887    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655921    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.655952    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.656018    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.656158    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b68886e-53d7-470d-8d43-893f733dae8f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.656314    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.656531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.656523    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.657326 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.656623    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.657326 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.656997    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.658206 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.658150    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b68886e-53d7-470d-8d43-893f733dae8f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.658336 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.658217    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.658679 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.658656    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.659108 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.658914    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b68886e-53d7-470d-8d43-893f733dae8f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.659108 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.659057    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.659248 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.659114    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.659248 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.659154    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf337cc-18f5-4451-88fe-351cf905dca8-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.659668 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.659651    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.680257 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.680227    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43892ef8-c4cc-414e-bc41-d1626001a0fa-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.680449 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.680365    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf337cc-18f5-4451-88fe-351cf905dca8-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.680449 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.680406    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4djd\" (UniqueName: \"kubernetes.io/projected/43892ef8-c4cc-414e-bc41-d1626001a0fa-kube-api-access-m4djd\") pod \"maas-default-gateway-openshift-default-6886b96f9-kbj7z\" (UID: \"43892ef8-c4cc-414e-bc41-d1626001a0fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.680584 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.680480    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b68886e-53d7-470d-8d43-893f733dae8f-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.680720 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.680698    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzlhg\" (UniqueName: \"kubernetes.io/projected/6b68886e-53d7-470d-8d43-893f733dae8f-kube-api-access-fzlhg\") pod \"maas-default-gateway-openshift-default-6886b96f9-4s5pw\" (UID: \"6b68886e-53d7-470d-8d43-893f733dae8f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.681183 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.681163    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qxz\" (UniqueName: \"kubernetes.io/projected/dcf337cc-18f5-4451-88fe-351cf905dca8-kube-api-access-95qxz\") pod \"maas-default-gateway-openshift-default-6886b96f9-8glk8\" (UID: \"dcf337cc-18f5-4451-88fe-351cf905dca8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.756547 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.756508    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565989d2-f1a9-4413-a50d-fd54da149761-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.756547 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.756554    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.756859 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.756591    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565989d2-f1a9-4413-a50d-fd54da149761-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.756859 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.756621    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.756859 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.756644    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.756859 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.756664    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565989d2-f1a9-4413-a50d-fd54da149761-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.756859 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.756811    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dg9d\" (UniqueName: \"kubernetes.io/projected/565989d2-f1a9-4413-a50d-fd54da149761-kube-api-access-9dg9d\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.757165 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.756869    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.757165 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.756893    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.757165 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.757119    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.757348 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.757181    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.757348 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.757311    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.757428 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.757339    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.757469 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.757457    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/565989d2-f1a9-4413-a50d-fd54da149761-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.759116 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.759095    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/565989d2-f1a9-4413-a50d-fd54da149761-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.759116 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.759104    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/565989d2-f1a9-4413-a50d-fd54da149761-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.781140 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.781051    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565989d2-f1a9-4413-a50d-fd54da149761-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.781269 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.781167    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dg9d\" (UniqueName: \"kubernetes.io/projected/565989d2-f1a9-4413-a50d-fd54da149761-kube-api-access-9dg9d\") pod \"maas-default-gateway-openshift-default-6886b96f9-thsm4\" (UID: \"565989d2-f1a9-4413-a50d-fd54da149761\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:27.841212 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.841172    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:27.866290 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.866248    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:27.890567 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.890532    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:27.893726 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:27.893693    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:28.160790 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:00:28.160745    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b68886e_53d7_470d_8d43_893f733dae8f.slice/crio-88a7f91a7b0c53990f402cd18c6736a5e9bf41e7f9ea4c957e8fdd26802d5d35 WatchSource:0}: Error finding container 88a7f91a7b0c53990f402cd18c6736a5e9bf41e7f9ea4c957e8fdd26802d5d35: Status 404 returned error can't find the container with id 88a7f91a7b0c53990f402cd18c6736a5e9bf41e7f9ea4c957e8fdd26802d5d35
Nov 02 15:00:28.161345 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.161311    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"]
Nov 02 15:00:28.162914 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.162881    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.163008 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.162941    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.163008 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.162973    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.200248 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.200181    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"]
Nov 02 15:00:28.201115 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:00:28.201089    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf337cc_18f5_4451_88fe_351cf905dca8.slice/crio-b81634ca89bac1e15cc91f7702a45e906c35429a4a46d78f9afef1399b24e97c WatchSource:0}: Error finding container b81634ca89bac1e15cc91f7702a45e906c35429a4a46d78f9afef1399b24e97c: Status 404 returned error can't find the container with id b81634ca89bac1e15cc91f7702a45e906c35429a4a46d78f9afef1399b24e97c
Nov 02 15:00:28.203590 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.203560    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.203677 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.203624    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.203677 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.203654    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.228830 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.228802    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"]
Nov 02 15:00:28.229862 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:00:28.229837    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43892ef8_c4cc_414e_bc41_d1626001a0fa.slice/crio-2c185cdb763f3445809c52a4162d7cac2b38666c44c0c475fcc1ba821debb2fa WatchSource:0}: Error finding container 2c185cdb763f3445809c52a4162d7cac2b38666c44c0c475fcc1ba821debb2fa: Status 404 returned error can't find the container with id 2c185cdb763f3445809c52a4162d7cac2b38666c44c0c475fcc1ba821debb2fa
Nov 02 15:00:28.233249 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.233213    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.233405 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.233319    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.233405 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.233372    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.237255 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.237231    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"]
Nov 02 15:00:28.242059 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:00:28.242036    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565989d2_f1a9_4413_a50d_fd54da149761.slice/crio-197d02bea012eb084af1409c25e6ada78bcb62a66cac0046ceb8dfb8f8f67ac8 WatchSource:0}: Error finding container 197d02bea012eb084af1409c25e6ada78bcb62a66cac0046ceb8dfb8f8f67ac8: Status 404 returned error can't find the container with id 197d02bea012eb084af1409c25e6ada78bcb62a66cac0046ceb8dfb8f8f67ac8
Nov 02 15:00:28.244955 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.244924    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.245076 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.245021    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.245076 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.245067    5034 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 02 15:00:28.436738 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.436639    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" event={"ID":"6b68886e-53d7-470d-8d43-893f733dae8f","Type":"ContainerStarted","Data":"1f66129e1fac694d3d244ae9a32cc019d36f32cca57a07c8b896004993065fdd"}
Nov 02 15:00:28.436738 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.436696    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" event={"ID":"6b68886e-53d7-470d-8d43-893f733dae8f","Type":"ContainerStarted","Data":"88a7f91a7b0c53990f402cd18c6736a5e9bf41e7f9ea4c957e8fdd26802d5d35"}
Nov 02 15:00:28.439097 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.439065    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" event={"ID":"dcf337cc-18f5-4451-88fe-351cf905dca8","Type":"ContainerStarted","Data":"3684fbef0b786325aca37ec153b4eccb215dd9471ccc382cb53879f648b4d3a6"}
Nov 02 15:00:28.439253 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.439114    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" event={"ID":"dcf337cc-18f5-4451-88fe-351cf905dca8","Type":"ContainerStarted","Data":"b81634ca89bac1e15cc91f7702a45e906c35429a4a46d78f9afef1399b24e97c"}
Nov 02 15:00:28.441298 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.441267    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" event={"ID":"565989d2-f1a9-4413-a50d-fd54da149761","Type":"ContainerStarted","Data":"ea4a9e7729be97e65fd53568267d3503a7ac2e2629f65ec30f0d131195cddd65"}
Nov 02 15:00:28.441440 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.441321    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" event={"ID":"565989d2-f1a9-4413-a50d-fd54da149761","Type":"ContainerStarted","Data":"197d02bea012eb084af1409c25e6ada78bcb62a66cac0046ceb8dfb8f8f67ac8"}
Nov 02 15:00:28.443194 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.443167    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" event={"ID":"43892ef8-c4cc-414e-bc41-d1626001a0fa","Type":"ContainerStarted","Data":"eb74c0c524ac4ef61449a89b999235b3594ec3d161f928727190728bc306600e"}
Nov 02 15:00:28.443312 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.443207    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" event={"ID":"43892ef8-c4cc-414e-bc41-d1626001a0fa","Type":"ContainerStarted","Data":"2c185cdb763f3445809c52a4162d7cac2b38666c44c0c475fcc1ba821debb2fa"}
Nov 02 15:00:28.485310 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.485250    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podStartSLOduration=1.485235542 podStartE2EDuration="1.485235542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:28.483595239 +0000 UTC m=+4619.234899717" watchObservedRunningTime="2026-11-02 15:00:28.485235542 +0000 UTC m=+4619.236539973"
Nov 02 15:00:28.582542 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.582472    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podStartSLOduration=1.58245047 podStartE2EDuration="1.58245047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:28.581676041 +0000 UTC m=+4619.332980474" watchObservedRunningTime="2026-11-02 15:00:28.58245047 +0000 UTC m=+4619.333754909"
Nov 02 15:00:28.583716 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.583647    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podStartSLOduration=1.583630614 podStartE2EDuration="1.583630614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:28.534293881 +0000 UTC m=+4619.285598320" watchObservedRunningTime="2026-11-02 15:00:28.583630614 +0000 UTC m=+4619.334935055"
Nov 02 15:00:28.631506 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.631444    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podStartSLOduration=1.631426182 podStartE2EDuration="1.631426182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:00:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:00:28.629241258 +0000 UTC m=+4619.380545695" watchObservedRunningTime="2026-11-02 15:00:28.631426182 +0000 UTC m=+4619.382730620"
Nov 02 15:00:28.843972 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.843875    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:28.867152 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.867109    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:28.895999 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.895869    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:28.895999 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:28.895961    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:29.846113 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:29.846069    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body=
Nov 02 15:00:29.846756 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:29.846723    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused"
Nov 02 15:00:29.870104 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:29.870066    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body=
Nov 02 15:00:29.870404 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:29.870372    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused"
Nov 02 15:00:29.899731 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:29.899695    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body=
Nov 02 15:00:29.900064 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:29.900032    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused"
Nov 02 15:00:29.900363 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:29.900343    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused" start-of-body=
Nov 02 15:00:29.900514 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:29.900495    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused"
Nov 02 15:00:30.842440 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:30.842394    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body=
Nov 02 15:00:30.842742 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:30.842697    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused"
Nov 02 15:00:30.869127 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:30.869090    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body=
Nov 02 15:00:30.869805 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:30.869739    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused"
Nov 02 15:00:30.892935 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:30.892897    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused" start-of-body=
Nov 02 15:00:30.893235 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:30.893205    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused"
Nov 02 15:00:30.897224 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:30.897193    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body=
Nov 02 15:00:30.897457 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:30.897432    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused"
Nov 02 15:00:31.936108 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:31.936058    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body=
Nov 02 15:00:31.936800 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:31.936730    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused"
Nov 02 15:00:31.937554 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:31.937525    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused" start-of-body=
Nov 02 15:00:31.940874 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:31.940840    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused"
Nov 02 15:00:31.962806 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:31.941013    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body=
Nov 02 15:00:31.963793 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:31.963736    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused"
Nov 02 15:00:31.964005 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:31.941610    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body=
Nov 02 15:00:31.964146 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:31.964125    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused"
Nov 02 15:00:32.845606 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:32.845561    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body=
Nov 02 15:00:32.846083 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:32.846052    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused"
Nov 02 15:00:32.867468 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:32.867431    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body=
Nov 02 15:00:32.867758 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:32.867715    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused"
Nov 02 15:00:32.905807 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:32.902527    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body=
Nov 02 15:00:32.905807 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:32.902595    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused"
Nov 02 15:00:32.905807 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:32.902677    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused" start-of-body=
Nov 02 15:00:32.905807 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:32.902697    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused"
Nov 02 15:00:33.843804 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:33.843616    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body=
Nov 02 15:00:33.843804 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:33.843678    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused"
Nov 02 15:00:33.874390 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:33.870268    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body=
Nov 02 15:00:33.874390 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:33.870361    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused"
Nov 02 15:00:33.897039 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:33.897000    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused" start-of-body=
Nov 02 15:00:33.897326 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:33.897300    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused"
Nov 02 15:00:33.897513 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:33.897497    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body=
Nov 02 15:00:33.897645 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:33.897627    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused"
Nov 02 15:00:34.847348 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:34.847303    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body=
Nov 02 15:00:34.848035 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:34.847981    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused"
Nov 02 15:00:34.867321 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:34.867279    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body=
Nov 02 15:00:34.867606 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:34.867579    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused"
Nov 02 15:00:34.895064 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:34.895017    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused" start-of-body=
Nov 02 15:00:34.895428 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:34.895402    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused"
Nov 02 15:00:34.895580 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:34.895345    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body=
Nov 02 15:00:34.895715 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:34.895696    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused"
Nov 02 15:00:35.841959 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:35.841907    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused" start-of-body=
Nov 02 15:00:35.842207 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:35.841985    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": dial tcp 10.132.1.182:15021: connect: connection refused"
Nov 02 15:00:35.867639 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:35.867584    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused" start-of-body=
Nov 02 15:00:35.868209 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:35.867656    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": dial tcp 10.132.1.183:15021: connect: connection refused"
Nov 02 15:00:35.891831 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:35.891529    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused" start-of-body=
Nov 02 15:00:35.891831 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:35.891594    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": dial tcp 10.132.1.185:15021: connect: connection refused"
Nov 02 15:00:35.894169 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:35.894141    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused" start-of-body=
Nov 02 15:00:35.894391 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:35.894364    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": dial tcp 10.132.1.184:15021: connect: connection refused"
Nov 02 15:00:37.797273 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.795420    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:37.797273 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.796107    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:37.798189 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.798169    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8"
Nov 02 15:00:37.838272 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.838238    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:37.839010 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.838982    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:37.839915 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.839888    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z"
Nov 02 15:00:37.842241 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.842212    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:00:37.842347 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.842265    5034 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:00:37.857543 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.857506    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:37.857960 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.857902    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:37.858985 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.858961    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw"
Nov 02 15:00:37.868257 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.868230    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:37.868542 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.868508    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:37.869185 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:37.869168    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4"
Nov 02 15:00:39.970450 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:39.970414    5034 scope.go:122] "RemoveContainer" containerID="f6048b0c07431c459b611b47cee369ab1ea1fac1a125fb7e006ec0a16c32998d"
Nov 02 15:00:39.979787 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:39.979753    5034 scope.go:122] "RemoveContainer" containerID="5086327a6ade5df3b49ca3fd504e2f92f7b2416a810af008adaff9cf64987235"
Nov 02 15:00:39.988731 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:39.988709    5034 scope.go:122] "RemoveContainer" containerID="96c1a2e1e65d8c5627895c86f93fb06b8a4917d686dfa09b206269c5ec3bceab"
Nov 02 15:00:39.997270 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:39.997245    5034 scope.go:122] "RemoveContainer" containerID="08852bc60cb728523fe56e60fd8717331d3751f369656b3bfa13e14f364aa9a4"
Nov 02 15:00:48.414527 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:00:48.414441    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:00:50.042587 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:50.042546    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"]
Nov 02 15:00:50.055449 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:50.055416    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"]
Nov 02 15:00:50.057715 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:50.057688    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ebde65-zhbgt"]
Nov 02 15:00:50.061508 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:50.061478    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47f066e1-6z6q9"]
Nov 02 15:00:51.393377 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:51.393338    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c" path="/var/lib/kubelet/pods/6453ba6f-29bc-4c77-bd61-7bfdbdd54c7c/volumes"
Nov 02 15:00:51.393840 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:00:51.393764    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c8386b-255e-4bea-bdef-17ef12b17d23" path="/var/lib/kubelet/pods/98c8386b-255e-4bea-bdef-17ef12b17d23/volumes"
Nov 02 15:01:00.041501 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:00.041462    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"]
Nov 02 15:01:00.043891 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:00.043838    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"]
Nov 02 15:01:00.046209 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:00.046183    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7bf411e-9plfn"]
Nov 02 15:01:00.048314 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:00.048290    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76cf25a4-5xghd"]
Nov 02 15:01:01.393553 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:01.393521    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556008cb-1991-4c8a-9030-d06d48b3dfd7" path="/var/lib/kubelet/pods/556008cb-1991-4c8a-9030-d06d48b3dfd7/volumes"
Nov 02 15:01:01.393996 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:01.393964    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f920e209-318d-45df-a17a-c97ee585197f" path="/var/lib/kubelet/pods/f920e209-318d-45df-a17a-c97ee585197f/volumes"
Nov 02 15:01:13.032195 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:13.032136    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"]
Nov 02 15:01:13.045613 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:13.045571    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ca422c6-s7cjv"]
Nov 02 15:01:13.398324 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:13.398222    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79598a5a-d187-43dd-9415-795f2d9867cd" path="/var/lib/kubelet/pods/79598a5a-d187-43dd-9415-795f2d9867cd/volumes"
Nov 02 15:01:18.579802 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:01:18.579302    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:01:23.032116 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:23.032073    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"]
Nov 02 15:01:23.044655 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:23.044617    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e75454d-bmwrp"]
Nov 02 15:01:23.416859 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:23.416705    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c921b24f-8d1a-471e-98c9-0c6ab4418857" path="/var/lib/kubelet/pods/c921b24f-8d1a-471e-98c9-0c6ab4418857/volumes"
Nov 02 15:01:34.029229 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:34.029192    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"]
Nov 02 15:01:34.042073 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:34.042043    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24611a13-n4qcq"]
Nov 02 15:01:35.393055 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:35.393021    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf3e04f-0208-4557-bb5d-39393d57ee04" path="/var/lib/kubelet/pods/9bf3e04f-0208-4557-bb5d-39393d57ee04/volumes"
Nov 02 15:01:45.037756 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:45.037716    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"]
Nov 02 15:01:45.050420 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:45.050384    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fa19b7a-69dk7"]
Nov 02 15:01:45.394027 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:01:45.393943    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d60b57-6e94-4c11-9128-f9668c124b9f" path="/var/lib/kubelet/pods/c0d60b57-6e94-4c11-9128-f9668c124b9f/volumes"
Nov 02 15:01:48.413720 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:01:48.413680    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:02:06.092803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.087825    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"]
Nov 02 15:02:06.294803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.294071    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"]
Nov 02 15:02:06.294803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.294244    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:06.344969 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.334300    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4711881d-kserve-self-signed-certs\""
Nov 02 15:02:06.344969 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.334449    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-4711881d-epp-sa-dockercfg-2lrj2\""
Nov 02 15:02:06.466820 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.466694    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c211b30f-97af-4542-bcaf-62d409e8cdf1-tls-certs\") pod \"e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg\" (UID: \"c211b30f-97af-4542-bcaf-62d409e8cdf1\") " pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:06.466820 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.466753    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kws6j\" (UniqueName: \"kubernetes.io/projected/c211b30f-97af-4542-bcaf-62d409e8cdf1-kube-api-access-kws6j\") pod \"e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg\" (UID: \"c211b30f-97af-4542-bcaf-62d409e8cdf1\") " pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:06.568163 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.568124    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c211b30f-97af-4542-bcaf-62d409e8cdf1-tls-certs\") pod \"e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg\" (UID: \"c211b30f-97af-4542-bcaf-62d409e8cdf1\") " pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:06.568491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.568452    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kws6j\" (UniqueName: \"kubernetes.io/projected/c211b30f-97af-4542-bcaf-62d409e8cdf1-kube-api-access-kws6j\") pod \"e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg\" (UID: \"c211b30f-97af-4542-bcaf-62d409e8cdf1\") " pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:06.598843 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.591703    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c211b30f-97af-4542-bcaf-62d409e8cdf1-tls-certs\") pod \"e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg\" (UID: \"c211b30f-97af-4542-bcaf-62d409e8cdf1\") " pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:06.599071 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.597073    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kws6j\" (UniqueName: \"kubernetes.io/projected/c211b30f-97af-4542-bcaf-62d409e8cdf1-kube-api-access-kws6j\") pod \"e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg\" (UID: \"c211b30f-97af-4542-bcaf-62d409e8cdf1\") " pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:06.732802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:06.731975    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:07.597149 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:07.597118    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"]
Nov 02 15:02:07.598380 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:07.598358    5034 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:02:07.615057 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:07.615006    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" event={"ID":"c211b30f-97af-4542-bcaf-62d409e8cdf1","Type":"ContainerStarted","Data":"a81511d1838227f313ca6d5fd7c41b840743b30118b34844f4f3bfc4301f45e4"}
Nov 02 15:02:12.701507 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:12.701469    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" event={"ID":"c211b30f-97af-4542-bcaf-62d409e8cdf1","Type":"ContainerStarted","Data":"1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be"}
Nov 02 15:02:12.701999 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:12.701614    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:12.746175 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:12.746123    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" podStartSLOduration=3.350955504 podStartE2EDuration="7.746110032s" totalImagesPullingTime="4.395154528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:02:12.745298162 +0000 UTC m=+4723.496602602" watchObservedRunningTime="2026-11-02 15:02:12.746110032 +0000 UTC m=+4723.497414500"
Nov 02 15:02:18.413026 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:02:18.412992    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:02:40.068765 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:40.068730    5034 scope.go:122] "RemoveContainer" containerID="996db8c6c6e98c61862a95b22a17c2aed54bdcd4dc8738bd2554f5eedfd4be07"
Nov 02 15:02:40.079120 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:40.079092    5034 scope.go:122] "RemoveContainer" containerID="d3377fde76f56f0a345fdae6b69f7376a38e1515bd61fcb55ceefcdefae045df"
Nov 02 15:02:40.088089 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:40.088066    5034 scope.go:122] "RemoveContainer" containerID="8f2167360a83f7554f4ee38e672c229e4087e6b2633f7a68d4abdf71a075814d"
Nov 02 15:02:40.096236 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:40.096207    5034 scope.go:122] "RemoveContainer" containerID="cdd80374fda154145b2a53c4f361666741b5f1475d69f7762384ca75aae1150a"
Nov 02 15:02:40.104199 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:40.104178    5034 scope.go:122] "RemoveContainer" containerID="4d10b4534952d985dfe743db8e4dcac674507dedadd9d4dded05c46a459aaed2"
Nov 02 15:02:40.112526 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:40.112497    5034 scope.go:122] "RemoveContainer" containerID="bd2740fd61d82b5af9d2c3a11dffc0debdc79f8e023f987a38ab9867c5603fd3"
Nov 02 15:02:40.121426 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:40.121397    5034 scope.go:122] "RemoveContainer" containerID="a0e3befede25eedfba05dd551a0596f7571c8219940bc379fd48c2df34f13d9b"
Nov 02 15:02:43.708341 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:02:43.708311    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:02:48.416987 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:02:48.416943    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:03:01.034830 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:01.034790    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"]
Nov 02 15:03:01.048576 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:01.048543    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3e2934-6w6dm"]
Nov 02 15:03:01.392959 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:01.392880    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136896ee-fa2e-4c92-9f6e-f2b18f3ce509" path="/var/lib/kubelet/pods/136896ee-fa2e-4c92-9f6e-f2b18f3ce509/volumes"
Nov 02 15:03:08.819253 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:08.819180    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"]
Nov 02 15:03:12.029653 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:12.029610    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"]
Nov 02 15:03:12.043513 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:12.043482    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4590b8df-nd5jv"]
Nov 02 15:03:13.473132 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:13.473076    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d3a8a6-4f39-4d48-9752-23a24b2a81a4" path="/var/lib/kubelet/pods/a2d3a8a6-4f39-4d48-9752-23a24b2a81a4/volumes"
Nov 02 15:03:13.719802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:13.715864    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:03:18.414351 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:03:18.414310    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:03:23.031217 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:23.031179    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"]
Nov 02 15:03:23.044574 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:23.044532    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d601ecfb-mrg9n"]
Nov 02 15:03:23.393443 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:23.393362    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896ac00f-5537-4e47-9356-c88ef5e1fbc4" path="/var/lib/kubelet/pods/896ac00f-5537-4e47-9356-c88ef5e1fbc4/volumes"
Nov 02 15:03:23.708108 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:23.708018    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:03:23.974445 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:23.974359    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerName="main" containerID="cri-o://1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be" gracePeriod=45
Nov 02 15:03:31.076970 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:31.076932    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:03:31.092820 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:31.092791    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:03:33.709066 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:33.709013    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:03:33.709565 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:33.709197    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:03:34.034116 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:34.034082    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"]
Nov 02 15:03:34.036553 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:34.036527    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93cadfad-92ggx"]
Nov 02 15:03:35.393944 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:35.393911    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c3756a5-7cbc-4241-b939-36bcc63204de" path="/var/lib/kubelet/pods/6c3756a5-7cbc-4241-b939-36bcc63204de/volumes"
Nov 02 15:03:40.167556 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:40.167523    5034 scope.go:122] "RemoveContainer" containerID="96bf5771ef3379f7f1c55989a095a2abff12749db51e2691c9d33886761a6413"
Nov 02 15:03:43.708640 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:43.708597    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:03:48.424542 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:03:48.424498    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:03:53.708754 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:53.708709    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:03:54.213517 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.213489    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:03:54.320178 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.320085    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c211b30f-97af-4542-bcaf-62d409e8cdf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/c211b30f-97af-4542-bcaf-62d409e8cdf1-tls-certs\") pod \"c211b30f-97af-4542-bcaf-62d409e8cdf1\" (UID: \"c211b30f-97af-4542-bcaf-62d409e8cdf1\") "
Nov 02 15:03:54.320357 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.320207    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c211b30f-97af-4542-bcaf-62d409e8cdf1-kube-api-access-kws6j\" (UniqueName: \"kubernetes.io/projected/c211b30f-97af-4542-bcaf-62d409e8cdf1-kube-api-access-kws6j\") pod \"c211b30f-97af-4542-bcaf-62d409e8cdf1\" (UID: \"c211b30f-97af-4542-bcaf-62d409e8cdf1\") "
Nov 02 15:03:54.322491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.322454    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c211b30f-97af-4542-bcaf-62d409e8cdf1-tls-certs" pod "c211b30f-97af-4542-bcaf-62d409e8cdf1" (UID: "c211b30f-97af-4542-bcaf-62d409e8cdf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:03:54.322606 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.322495    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c211b30f-97af-4542-bcaf-62d409e8cdf1-kube-api-access-kws6j" pod "c211b30f-97af-4542-bcaf-62d409e8cdf1" (UID: "c211b30f-97af-4542-bcaf-62d409e8cdf1"). InnerVolumeSpecName "kube-api-access-kws6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:03:54.410358 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.410319    5034 generic.go:356] "Generic (PLEG): container finished" podID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerID="1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be" exitCode=0
Nov 02 15:03:54.410558 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.410369    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"
Nov 02 15:03:54.410558 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.410397    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" event={"ID":"c211b30f-97af-4542-bcaf-62d409e8cdf1","Type":"ContainerDied","Data":"1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be"}
Nov 02 15:03:54.410558 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.410450    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg" event={"ID":"c211b30f-97af-4542-bcaf-62d409e8cdf1","Type":"ContainerDied","Data":"a81511d1838227f313ca6d5fd7c41b840743b30118b34844f4f3bfc4301f45e4"}
Nov 02 15:03:54.410558 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.410472    5034 scope.go:122] "RemoveContainer" containerID="1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be"
Nov 02 15:03:54.421162 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.421137    5034 scope.go:122] "RemoveContainer" containerID="1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be"
Nov 02 15:03:54.421275 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.421201    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c211b30f-97af-4542-bcaf-62d409e8cdf1-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:03:54.421275 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.421223    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kws6j\" (UniqueName: \"kubernetes.io/projected/c211b30f-97af-4542-bcaf-62d409e8cdf1-kube-api-access-kws6j\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:03:54.421497 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:03:54.421479    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be\": container with ID starting with 1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be not found: ID does not exist" containerID="1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be"
Nov 02 15:03:54.421534 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.421506    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be"} err="failed to get container status \"1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be\": rpc error: code = NotFound desc = could not find container \"1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be\": container with ID starting with 1e97ca823f874d630600332f9afb2b6d87c9595be882eee854f271100e8894be not found: ID does not exist"
Nov 02 15:03:54.466517 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.466480    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"]
Nov 02 15:03:54.470331 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:54.470301    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-4711881d-kserve-router-scheduler-886bd7dfb-jxrrg"]
Nov 02 15:03:55.393390 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:03:55.393358    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" path="/var/lib/kubelet/pods/c211b30f-97af-4542-bcaf-62d409e8cdf1/volumes"
Nov 02 15:04:18.413201 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:04:18.413162    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:04:19.246267 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.246227    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"]
Nov 02 15:04:19.246707 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.246693    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerName="main"
Nov 02 15:04:19.246756 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.246708    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerName="main"
Nov 02 15:04:19.246849 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.246822    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c211b30f-97af-4542-bcaf-62d409e8cdf1" containerName="main"
Nov 02 15:04:19.249884 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.249861    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.268824 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.268791    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:04:19.268986 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.268893    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:04:19.273510 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.273471    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa65da05-f6tbx\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.273698 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.273580    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa65da05-f6tbx\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.273812 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.273696    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt8j2\" (UniqueName: \"kubernetes.io/projected/bf61bf0f-3b68-46df-bf5e-507806ac85b2-kube-api-access-tt8j2\") pod \"maas-api-revoke-sub-fa65da05-f6tbx\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.282113 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.282083    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:04:19.282504 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.282489    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:04:19.285748 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.285719    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"]
Nov 02 15:04:19.374556 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.374521    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa65da05-f6tbx\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.374716 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.374587    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa65da05-f6tbx\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.374716 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.374654    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt8j2\" (UniqueName: \"kubernetes.io/projected/bf61bf0f-3b68-46df-bf5e-507806ac85b2-kube-api-access-tt8j2\") pod \"maas-api-revoke-sub-fa65da05-f6tbx\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.375355 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.375328    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa65da05-f6tbx\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.376977 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.376958    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa65da05-f6tbx\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.400672 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.400633    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt8j2\" (UniqueName: \"kubernetes.io/projected/bf61bf0f-3b68-46df-bf5e-507806ac85b2-kube-api-access-tt8j2\") pod \"maas-api-revoke-sub-fa65da05-f6tbx\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.564694 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.564618    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:19.867607 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:04:19.867575    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf61bf0f_3b68_46df_bf5e_507806ac85b2.slice/crio-ff062b37a46b49afd91606b0a772360dd5c7ddc552554247b91ec40fb89693de WatchSource:0}: Error finding container ff062b37a46b49afd91606b0a772360dd5c7ddc552554247b91ec40fb89693de: Status 404 returned error can't find the container with id ff062b37a46b49afd91606b0a772360dd5c7ddc552554247b91ec40fb89693de
Nov 02 15:04:19.867928 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:19.867901    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"]
Nov 02 15:04:20.556446 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:20.556403    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx" event={"ID":"bf61bf0f-3b68-46df-bf5e-507806ac85b2","Type":"ContainerStarted","Data":"0fb127da8e47affabe75a58bdfa7f704abd6d163b676d404e80c4d8424d55c20"}
Nov 02 15:04:20.556641 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:20.556456    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx" event={"ID":"bf61bf0f-3b68-46df-bf5e-507806ac85b2","Type":"ContainerStarted","Data":"ff062b37a46b49afd91606b0a772360dd5c7ddc552554247b91ec40fb89693de"}
Nov 02 15:04:20.602362 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:20.602294    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx" podStartSLOduration=1.6022758910000001 podStartE2EDuration="1.602275891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:20.601952202 +0000 UTC m=+4851.353256642" watchObservedRunningTime="2026-11-02 15:04:20.602275891 +0000 UTC m=+4851.353580330"
Nov 02 15:04:21.565168 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:21.565124    5034 generic.go:356] "Generic (PLEG): container finished" podID="bf61bf0f-3b68-46df-bf5e-507806ac85b2" containerID="0fb127da8e47affabe75a58bdfa7f704abd6d163b676d404e80c4d8424d55c20" exitCode=0
Nov 02 15:04:21.565382 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:21.565172    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx" event={"ID":"bf61bf0f-3b68-46df-bf5e-507806ac85b2","Type":"ContainerDied","Data":"0fb127da8e47affabe75a58bdfa7f704abd6d163b676d404e80c4d8424d55c20"}
Nov 02 15:04:22.735864 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.735833    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:22.810710 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.810674    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf61bf0f-3b68-46df-bf5e-507806ac85b2-kube-api-access-tt8j2\" (UniqueName: \"kubernetes.io/projected/bf61bf0f-3b68-46df-bf5e-507806ac85b2-kube-api-access-tt8j2\") pod \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") "
Nov 02 15:04:22.810710 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.810714    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-openshift-ca\") pod \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") "
Nov 02 15:04:22.811008 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.810748    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-xks-ca\") pod \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\" (UID: \"bf61bf0f-3b68-46df-bf5e-507806ac85b2\") "
Nov 02 15:04:22.811205 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.811172    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-openshift-ca" pod "bf61bf0f-3b68-46df-bf5e-507806ac85b2" (UID: "bf61bf0f-3b68-46df-bf5e-507806ac85b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:04:22.813021 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.812993    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-xks-ca" pod "bf61bf0f-3b68-46df-bf5e-507806ac85b2" (UID: "bf61bf0f-3b68-46df-bf5e-507806ac85b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:04:22.813124 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.813019    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf61bf0f-3b68-46df-bf5e-507806ac85b2-kube-api-access-tt8j2" pod "bf61bf0f-3b68-46df-bf5e-507806ac85b2" (UID: "bf61bf0f-3b68-46df-bf5e-507806ac85b2"). InnerVolumeSpecName "kube-api-access-tt8j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:04:22.912000 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.911909    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt8j2\" (UniqueName: \"kubernetes.io/projected/bf61bf0f-3b68-46df-bf5e-507806ac85b2-kube-api-access-tt8j2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:04:22.912000 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.911938    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:04:22.912000 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:22.911947    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf61bf0f-3b68-46df-bf5e-507806ac85b2-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:04:23.577729 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:23.577698    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"
Nov 02 15:04:23.577950 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:23.577738    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx" event={"ID":"bf61bf0f-3b68-46df-bf5e-507806ac85b2","Type":"ContainerDied","Data":"ff062b37a46b49afd91606b0a772360dd5c7ddc552554247b91ec40fb89693de"}
Nov 02 15:04:23.577950 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:23.577809    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff062b37a46b49afd91606b0a772360dd5c7ddc552554247b91ec40fb89693de"
Nov 02 15:04:40.189641 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:40.189609    5034 scope.go:122] "RemoveContainer" containerID="b97fd39238fda2936b81cb853cec8f074a33a33e3abdd1bdd4785092e89de194"
Nov 02 15:04:40.198496 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:40.198473    5034 scope.go:122] "RemoveContainer" containerID="0dfd6792719a7ff859ec2a6d5578f8c59fb747de8da9cc828260d831cc390dce"
Nov 02 15:04:40.206743 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:40.206724    5034 scope.go:122] "RemoveContainer" containerID="06ef77ce75f03c7d215b694ecfb7ced64d7e81b6abf917b07911e621afa6be77"
Nov 02 15:04:40.214345 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:04:40.214327    5034 scope.go:122] "RemoveContainer" containerID="3b754a05e5e703a1c83be296c7278db869564c07cde07756758400836015d3e2"
Nov 02 15:04:48.412752 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:04:48.412719    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:05:11.650681 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.650587    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"]
Nov 02 15:05:11.651328 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.651284    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bf61bf0f-3b68-46df-bf5e-507806ac85b2" containerName="revoke-keys"
Nov 02 15:05:11.651328 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.651305    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf61bf0f-3b68-46df-bf5e-507806ac85b2" containerName="revoke-keys"
Nov 02 15:05:11.651464 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.651414    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf61bf0f-3b68-46df-bf5e-507806ac85b2" containerName="revoke-keys"
Nov 02 15:05:11.654756 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.654733    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:11.662423 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.662396    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"]
Nov 02 15:05:11.665946 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.665930    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:11.676459 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.676433    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxwvt\" (UniqueName: \"kubernetes.io/projected/4110afc3-3118-47ea-bc83-40273b8c76e0-kube-api-access-nxwvt\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv\" (UID: \"4110afc3-3118-47ea-bc83-40273b8c76e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:11.676577 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.676490    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zsc\" (UniqueName: \"kubernetes.io/projected/af5477c2-d75f-4a92-b6bf-a80e51186c9b-kube-api-access-x2zsc\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg\" (UID: \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:11.676702 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.676675    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4110afc3-3118-47ea-bc83-40273b8c76e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv\" (UID: \"4110afc3-3118-47ea-bc83-40273b8c76e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:11.676765 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.676723    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5477c2-d75f-4a92-b6bf-a80e51186c9b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg\" (UID: \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:11.676765 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.676734    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"]
Nov 02 15:05:11.686279 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.686259    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6e5c6f\""
Nov 02 15:05:11.686396 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.686301    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a7e7622c-dockercfg-gmt89\""
Nov 02 15:05:11.721636 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.721605    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"]
Nov 02 15:05:11.777916 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.777881    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxwvt\" (UniqueName: \"kubernetes.io/projected/4110afc3-3118-47ea-bc83-40273b8c76e0-kube-api-access-nxwvt\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv\" (UID: \"4110afc3-3118-47ea-bc83-40273b8c76e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:11.778093 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.777934    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zsc\" (UniqueName: \"kubernetes.io/projected/af5477c2-d75f-4a92-b6bf-a80e51186c9b-kube-api-access-x2zsc\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg\" (UID: \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:11.778093 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.778001    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4110afc3-3118-47ea-bc83-40273b8c76e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv\" (UID: \"4110afc3-3118-47ea-bc83-40273b8c76e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:11.778093 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.778022    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5477c2-d75f-4a92-b6bf-a80e51186c9b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg\" (UID: \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:11.778822 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.778773    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4110afc3-3118-47ea-bc83-40273b8c76e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv\" (UID: \"4110afc3-3118-47ea-bc83-40273b8c76e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:11.778890 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.778841    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5477c2-d75f-4a92-b6bf-a80e51186c9b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg\" (UID: \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:11.810996 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.810963    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zsc\" (UniqueName: \"kubernetes.io/projected/af5477c2-d75f-4a92-b6bf-a80e51186c9b-kube-api-access-x2zsc\") pod \"payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg\" (UID: \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:11.814618 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.814593    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxwvt\" (UniqueName: \"kubernetes.io/projected/4110afc3-3118-47ea-bc83-40273b8c76e0-kube-api-access-nxwvt\") pod \"payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv\" (UID: \"4110afc3-3118-47ea-bc83-40273b8c76e0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:11.968840 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.968743    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:11.980277 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:11.980254    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:12.278016 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:05:12.277977    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5477c2_d75f_4a92_b6bf_a80e51186c9b.slice/crio-6542c4e2562b36ddd066bbbe5b28cd36cf436e4f8e47b72ee21754c65c5d76e9 WatchSource:0}: Error finding container 6542c4e2562b36ddd066bbbe5b28cd36cf436e4f8e47b72ee21754c65c5d76e9: Status 404 returned error can't find the container with id 6542c4e2562b36ddd066bbbe5b28cd36cf436e4f8e47b72ee21754c65c5d76e9
Nov 02 15:05:12.279369 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.279345    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"]
Nov 02 15:05:12.300927 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.300890    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"]
Nov 02 15:05:12.309955 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:05:12.309929    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4110afc3_3118_47ea_bc83_40273b8c76e0.slice/crio-ac7a9f13521202d9be095524aa6b40487395bc6f049cedcc178c783ace56cfdc WatchSource:0}: Error finding container ac7a9f13521202d9be095524aa6b40487395bc6f049cedcc178c783ace56cfdc: Status 404 returned error can't find the container with id ac7a9f13521202d9be095524aa6b40487395bc6f049cedcc178c783ace56cfdc
Nov 02 15:05:12.842291 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.842259    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg" event={"ID":"af5477c2-d75f-4a92-b6bf-a80e51186c9b","Type":"ContainerStarted","Data":"c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349"}
Nov 02 15:05:12.842751 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.842327    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:12.842751 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.842353    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg" event={"ID":"af5477c2-d75f-4a92-b6bf-a80e51186c9b","Type":"ContainerStarted","Data":"6542c4e2562b36ddd066bbbe5b28cd36cf436e4f8e47b72ee21754c65c5d76e9"}
Nov 02 15:05:12.843878 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.843847    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv" event={"ID":"4110afc3-3118-47ea-bc83-40273b8c76e0","Type":"ContainerStarted","Data":"2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff"}
Nov 02 15:05:12.844012 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.843899    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv" event={"ID":"4110afc3-3118-47ea-bc83-40273b8c76e0","Type":"ContainerStarted","Data":"ac7a9f13521202d9be095524aa6b40487395bc6f049cedcc178c783ace56cfdc"}
Nov 02 15:05:12.844012 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.843937    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:12.887047 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.886984    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg" podStartSLOduration=1.886972342 podStartE2EDuration="1.886972342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:12.885373384 +0000 UTC m=+4903.636677834" watchObservedRunningTime="2026-11-02 15:05:12.886972342 +0000 UTC m=+4903.638276777"
Nov 02 15:05:12.930114 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:12.930060    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv" podStartSLOduration=1.9300451010000002 podStartE2EDuration="1.930045101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:12.928147846 +0000 UTC m=+4903.679452285" watchObservedRunningTime="2026-11-02 15:05:12.930045101 +0000 UTC m=+4903.681349541"
Nov 02 15:05:18.415676 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:05:18.415622    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:05:19.433249 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.433217    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"]
Nov 02 15:05:19.436599 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.436579    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.459183 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.459144    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.459355 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.459190    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hcr\" (UniqueName: \"kubernetes.io/projected/9b80f113-3804-4859-a339-55f6796d25f2-kube-api-access-r5hcr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.459355 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.459330    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.468403 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.468366    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:05:19.468403 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.468366    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:05:19.468668 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.468491    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:05:19.468668 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.468501    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:05:19.470757 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.470729    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"]
Nov 02 15:05:19.560853 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.560819    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.561084 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.560857    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hcr\" (UniqueName: \"kubernetes.io/projected/9b80f113-3804-4859-a339-55f6796d25f2-kube-api-access-r5hcr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.561084 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.560919    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.561631 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.561604    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.563255 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.563232    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.586537 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.586506    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hcr\" (UniqueName: \"kubernetes.io/projected/9b80f113-3804-4859-a339-55f6796d25f2-kube-api-access-r5hcr\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:19.750405 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:19.750369    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:20.056968 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:20.056938    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"]
Nov 02 15:05:20.057692 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:05:20.057669    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b80f113_3804_4859_a339_55f6796d25f2.slice/crio-c324c2b0ab8175c79b4d60b31663525ee0243bd7fff94caeddde2606a1b9a303 WatchSource:0}: Error finding container c324c2b0ab8175c79b4d60b31663525ee0243bd7fff94caeddde2606a1b9a303: Status 404 returned error can't find the container with id c324c2b0ab8175c79b4d60b31663525ee0243bd7fff94caeddde2606a1b9a303
Nov 02 15:05:20.889135 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:20.889100    5034 generic.go:356] "Generic (PLEG): container finished" podID="9b80f113-3804-4859-a339-55f6796d25f2" containerID="d29c680c1a0c11b72606fb06df2a32c27c65a24b10f937b9810eeec171904031" exitCode=0
Nov 02 15:05:20.889599 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:20.889183    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc" event={"ID":"9b80f113-3804-4859-a339-55f6796d25f2","Type":"ContainerDied","Data":"d29c680c1a0c11b72606fb06df2a32c27c65a24b10f937b9810eeec171904031"}
Nov 02 15:05:20.889599 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:20.889234    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc" event={"ID":"9b80f113-3804-4859-a339-55f6796d25f2","Type":"ContainerStarted","Data":"c324c2b0ab8175c79b4d60b31663525ee0243bd7fff94caeddde2606a1b9a303"}
Nov 02 15:05:22.028194 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.028167    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:22.086813 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.086710    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b80f113-3804-4859-a339-55f6796d25f2-kube-api-access-r5hcr\" (UniqueName: \"kubernetes.io/projected/9b80f113-3804-4859-a339-55f6796d25f2-kube-api-access-r5hcr\") pod \"9b80f113-3804-4859-a339-55f6796d25f2\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") "
Nov 02 15:05:22.086813 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.086761    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-xks-ca\") pod \"9b80f113-3804-4859-a339-55f6796d25f2\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") "
Nov 02 15:05:22.087068 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.086885    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-openshift-ca\") pod \"9b80f113-3804-4859-a339-55f6796d25f2\" (UID: \"9b80f113-3804-4859-a339-55f6796d25f2\") "
Nov 02 15:05:22.087362 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.087326    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-openshift-ca" pod "9b80f113-3804-4859-a339-55f6796d25f2" (UID: "9b80f113-3804-4859-a339-55f6796d25f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:05:22.089094 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.089064    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-xks-ca" pod "9b80f113-3804-4859-a339-55f6796d25f2" (UID: "9b80f113-3804-4859-a339-55f6796d25f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:05:22.089237 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.089115    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b80f113-3804-4859-a339-55f6796d25f2-kube-api-access-r5hcr" pod "9b80f113-3804-4859-a339-55f6796d25f2" (UID: "9b80f113-3804-4859-a339-55f6796d25f2"). InnerVolumeSpecName "kube-api-access-r5hcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:05:22.188810 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.188733    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:05:22.188810 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.188769    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hcr\" (UniqueName: \"kubernetes.io/projected/9b80f113-3804-4859-a339-55f6796d25f2-kube-api-access-r5hcr\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:05:22.188810 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.188821    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b80f113-3804-4859-a339-55f6796d25f2-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:05:22.900647 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.900619    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"
Nov 02 15:05:22.900870 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.900659    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc" event={"ID":"9b80f113-3804-4859-a339-55f6796d25f2","Type":"ContainerDied","Data":"c324c2b0ab8175c79b4d60b31663525ee0243bd7fff94caeddde2606a1b9a303"}
Nov 02 15:05:22.900870 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:22.900707    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c324c2b0ab8175c79b4d60b31663525ee0243bd7fff94caeddde2606a1b9a303"
Nov 02 15:05:23.853014 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:23.852986    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:23.853433 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:23.853056    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:25.037978 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:25.037944    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"]
Nov 02 15:05:25.051447 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:25.051415    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-z2pmm"]
Nov 02 15:05:25.393209 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:25.393133    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34051f1-43e5-4b16-82e6-60d98e539607" path="/var/lib/kubelet/pods/c34051f1-43e5-4b16-82e6-60d98e539607/volumes"
Nov 02 15:05:33.139695 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:33.133374    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"]
Nov 02 15:05:33.170361 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:33.170328    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"]
Nov 02 15:05:34.956207 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:34.956167    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"]
Nov 02 15:05:34.970101 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:34.970072    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6e5c6f-fn8xc"]
Nov 02 15:05:35.394349 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:35.394306    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b80f113-3804-4859-a339-55f6796d25f2" path="/var/lib/kubelet/pods/9b80f113-3804-4859-a339-55f6796d25f2/volumes"
Nov 02 15:05:38.134699 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.134636    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg" podUID="af5477c2-d75f-4a92-b6bf-a80e51186c9b" containerName="payload-processing" containerID="cri-o://c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349" gracePeriod=25
Nov 02 15:05:38.171343 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.171308    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv" podUID="4110afc3-3118-47ea-bc83-40273b8c76e0" containerName="payload-pre-processing" containerID="cri-o://2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff" gracePeriod=25
Nov 02 15:05:38.438417 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.438387    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:38.461385 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.461359    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:38.558481 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.558447    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5477c2-d75f-4a92-b6bf-a80e51186c9b-kube-api-access-x2zsc\" (UniqueName: \"kubernetes.io/projected/af5477c2-d75f-4a92-b6bf-a80e51186c9b-kube-api-access-x2zsc\") pod \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\" (UID: \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\") "
Nov 02 15:05:38.558656 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.558564    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4110afc3-3118-47ea-bc83-40273b8c76e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4110afc3-3118-47ea-bc83-40273b8c76e0-plugins-config-volume\") pod \"4110afc3-3118-47ea-bc83-40273b8c76e0\" (UID: \"4110afc3-3118-47ea-bc83-40273b8c76e0\") "
Nov 02 15:05:38.558656 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.558628    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4110afc3-3118-47ea-bc83-40273b8c76e0-kube-api-access-nxwvt\" (UniqueName: \"kubernetes.io/projected/4110afc3-3118-47ea-bc83-40273b8c76e0-kube-api-access-nxwvt\") pod \"4110afc3-3118-47ea-bc83-40273b8c76e0\" (UID: \"4110afc3-3118-47ea-bc83-40273b8c76e0\") "
Nov 02 15:05:38.558874 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.558723    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5477c2-d75f-4a92-b6bf-a80e51186c9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5477c2-d75f-4a92-b6bf-a80e51186c9b-plugins-config-volume\") pod \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\" (UID: \"af5477c2-d75f-4a92-b6bf-a80e51186c9b\") "
Nov 02 15:05:38.559119 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.559082    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4110afc3-3118-47ea-bc83-40273b8c76e0-plugins-config-volume" pod "4110afc3-3118-47ea-bc83-40273b8c76e0" (UID: "4110afc3-3118-47ea-bc83-40273b8c76e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:05:38.559282 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.559239    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5477c2-d75f-4a92-b6bf-a80e51186c9b-plugins-config-volume" pod "af5477c2-d75f-4a92-b6bf-a80e51186c9b" (UID: "af5477c2-d75f-4a92-b6bf-a80e51186c9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:05:38.561236 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.561214    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5477c2-d75f-4a92-b6bf-a80e51186c9b-kube-api-access-x2zsc" pod "af5477c2-d75f-4a92-b6bf-a80e51186c9b" (UID: "af5477c2-d75f-4a92-b6bf-a80e51186c9b"). InnerVolumeSpecName "kube-api-access-x2zsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:05:38.561313 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.561245    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4110afc3-3118-47ea-bc83-40273b8c76e0-kube-api-access-nxwvt" pod "4110afc3-3118-47ea-bc83-40273b8c76e0" (UID: "4110afc3-3118-47ea-bc83-40273b8c76e0"). InnerVolumeSpecName "kube-api-access-nxwvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:05:38.660445 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.660358    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4110afc3-3118-47ea-bc83-40273b8c76e0-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:05:38.660445 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.660390    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxwvt\" (UniqueName: \"kubernetes.io/projected/4110afc3-3118-47ea-bc83-40273b8c76e0-kube-api-access-nxwvt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:05:38.660445 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.660400    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af5477c2-d75f-4a92-b6bf-a80e51186c9b-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:05:38.660445 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.660409    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2zsc\" (UniqueName: \"kubernetes.io/projected/af5477c2-d75f-4a92-b6bf-a80e51186c9b-kube-api-access-x2zsc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:05:38.988644 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.988605    5034 generic.go:356] "Generic (PLEG): container finished" podID="4110afc3-3118-47ea-bc83-40273b8c76e0" containerID="2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff" exitCode=0
Nov 02 15:05:38.988889 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.988660    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"
Nov 02 15:05:38.988889 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.988689    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv" event={"ID":"4110afc3-3118-47ea-bc83-40273b8c76e0","Type":"ContainerDied","Data":"2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff"}
Nov 02 15:05:38.988889 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.988750    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv" event={"ID":"4110afc3-3118-47ea-bc83-40273b8c76e0","Type":"ContainerDied","Data":"ac7a9f13521202d9be095524aa6b40487395bc6f049cedcc178c783ace56cfdc"}
Nov 02 15:05:38.988889 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.988798    5034 scope.go:122] "RemoveContainer" containerID="2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff"
Nov 02 15:05:38.990229 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.990201    5034 generic.go:356] "Generic (PLEG): container finished" podID="af5477c2-d75f-4a92-b6bf-a80e51186c9b" containerID="c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349" exitCode=0
Nov 02 15:05:38.990353 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.990245    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"
Nov 02 15:05:38.990353 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.990255    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg" event={"ID":"af5477c2-d75f-4a92-b6bf-a80e51186c9b","Type":"ContainerDied","Data":"c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349"}
Nov 02 15:05:38.990353 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.990298    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg" event={"ID":"af5477c2-d75f-4a92-b6bf-a80e51186c9b","Type":"ContainerDied","Data":"6542c4e2562b36ddd066bbbe5b28cd36cf436e4f8e47b72ee21754c65c5d76e9"}
Nov 02 15:05:38.999175 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.999156    5034 scope.go:122] "RemoveContainer" containerID="2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff"
Nov 02 15:05:38.999511 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:05:38.999485    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff\": container with ID starting with 2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff not found: ID does not exist" containerID="2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff"
Nov 02 15:05:38.999592 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.999524    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff"} err="failed to get container status \"2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff\": rpc error: code = NotFound desc = could not find container \"2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff\": container with ID starting with 2e06567c6b418762b10ee73e68cc427c265c1ce7b8e393874afa1861526131ff not found: ID does not exist"
Nov 02 15:05:38.999592 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:38.999550    5034 scope.go:122] "RemoveContainer" containerID="c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349"
Nov 02 15:05:39.008281 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:39.008260    5034 scope.go:122] "RemoveContainer" containerID="c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349"
Nov 02 15:05:39.008569 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:05:39.008545    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349\": container with ID starting with c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349 not found: ID does not exist" containerID="c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349"
Nov 02 15:05:39.008636 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:39.008578    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349"} err="failed to get container status \"c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349\": rpc error: code = NotFound desc = could not find container \"c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349\": container with ID starting with c79c14b5a67840d368e329bbc1a90f06bf7a1bdfe4b90c824a572692f0c8c349 not found: ID does not exist"
Nov 02 15:05:39.045994 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:39.045957    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"]
Nov 02 15:05:39.058285 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:39.058254    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6e5c6f-759d46777ksvcg"]
Nov 02 15:05:39.087654 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:39.087589    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"]
Nov 02 15:05:39.091860 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:39.091830    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6e5c6f-57f97kz4kv"]
Nov 02 15:05:39.394477 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:39.394404    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4110afc3-3118-47ea-bc83-40273b8c76e0" path="/var/lib/kubelet/pods/4110afc3-3118-47ea-bc83-40273b8c76e0/volumes"
Nov 02 15:05:39.394859 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:05:39.394821    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5477c2-d75f-4a92-b6bf-a80e51186c9b" path="/var/lib/kubelet/pods/af5477c2-d75f-4a92-b6bf-a80e51186c9b/volumes"
Nov 02 15:05:48.413369 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:05:48.413329    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:06:06.327232 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.327189    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"]
Nov 02 15:06:06.327998 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.327975    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4110afc3-3118-47ea-bc83-40273b8c76e0" containerName="payload-pre-processing"
Nov 02 15:06:06.328054 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.328000    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4110afc3-3118-47ea-bc83-40273b8c76e0" containerName="payload-pre-processing"
Nov 02 15:06:06.328054 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.328036    5034 cpu_manager.go:414] "containerMap: removing container" podUID="af5477c2-d75f-4a92-b6bf-a80e51186c9b" containerName="payload-processing"
Nov 02 15:06:06.328054 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.328044    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5477c2-d75f-4a92-b6bf-a80e51186c9b" containerName="payload-processing"
Nov 02 15:06:06.328164 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.328065    5034 cpu_manager.go:414] "containerMap: removing container" podUID="9b80f113-3804-4859-a339-55f6796d25f2" containerName="revoke-keys"
Nov 02 15:06:06.328164 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.328072    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b80f113-3804-4859-a339-55f6796d25f2" containerName="revoke-keys"
Nov 02 15:06:06.328234 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.328198    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5477c2-d75f-4a92-b6bf-a80e51186c9b" containerName="payload-processing"
Nov 02 15:06:06.328234 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.328211    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b80f113-3804-4859-a339-55f6796d25f2" containerName="revoke-keys"
Nov 02 15:06:06.328234 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.328229    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4110afc3-3118-47ea-bc83-40273b8c76e0" containerName="payload-pre-processing"
Nov 02 15:06:06.332023 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.331997    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.338797 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.338757    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"]
Nov 02 15:06:06.350604 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.350581    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7540119b-kserve-self-signed-certs\""
Nov 02 15:06:06.400583 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.400551    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-home\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.400725 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.400585    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh6l8\" (UniqueName: \"kubernetes.io/projected/deb89a97-000d-4a7f-80aa-9c4c00b45990-kube-api-access-mh6l8\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.400725 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.400613    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-tmp-dir\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.400848 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.400769    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb89a97-000d-4a7f-80aa-9c4c00b45990-tls-certs\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.400848 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.400837    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-model-cache\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.400994 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.400970    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-dshm\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.501743 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.501696    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-home\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.502132 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.502104    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh6l8\" (UniqueName: \"kubernetes.io/projected/deb89a97-000d-4a7f-80aa-9c4c00b45990-kube-api-access-mh6l8\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.502304 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.502287    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-tmp-dir\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.502571 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.502556    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb89a97-000d-4a7f-80aa-9c4c00b45990-tls-certs\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.502740 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.502725    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-model-cache\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.503238 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.503206    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-model-cache\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.503577 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.503541    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-tmp-dir\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.503577 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.503552    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-home\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.503822 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.503735    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-dshm\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.506979 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.506954    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-dshm\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.507948 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.507920    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb89a97-000d-4a7f-80aa-9c4c00b45990-tls-certs\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.528417 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.528389    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh6l8\" (UniqueName: \"kubernetes.io/projected/deb89a97-000d-4a7f-80aa-9c4c00b45990-kube-api-access-mh6l8\") pod \"e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") " pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.648521 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.648428    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:06.949466 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:06:06.949437    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeb89a97_000d_4a7f_80aa_9c4c00b45990.slice/crio-59780f2bb5620a01151bb5a8b9a7d5b77bbcc54072ef519f0c4b5c22110ce09b WatchSource:0}: Error finding container 59780f2bb5620a01151bb5a8b9a7d5b77bbcc54072ef519f0c4b5c22110ce09b: Status 404 returned error can't find the container with id 59780f2bb5620a01151bb5a8b9a7d5b77bbcc54072ef519f0c4b5c22110ce09b
Nov 02 15:06:06.949601 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:06.949545    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"]
Nov 02 15:06:07.185366 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:07.185331    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz" event={"ID":"deb89a97-000d-4a7f-80aa-9c4c00b45990","Type":"ContainerStarted","Data":"0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac"}
Nov 02 15:06:07.185366 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:07.185378    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz" event={"ID":"deb89a97-000d-4a7f-80aa-9c4c00b45990","Type":"ContainerStarted","Data":"59780f2bb5620a01151bb5a8b9a7d5b77bbcc54072ef519f0c4b5c22110ce09b"}
Nov 02 15:06:07.232721 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:07.232677    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz" podStartSLOduration=1.232663651 podStartE2EDuration="1.232663651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:06:07.230051196 +0000 UTC m=+4957.981355635" watchObservedRunningTime="2026-11-02 15:06:07.232663651 +0000 UTC m=+4957.983968088"
Nov 02 15:06:07.649196 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:07.649114    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:16.649198 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:16.649150    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:16.662112 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:16.662078    5034 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:17.285854 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:17.285823    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:18.414907 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:06:18.414860    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:06:37.143771 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:37.143734    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"]
Nov 02 15:06:40.265532 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:40.265493    5034 scope.go:122] "RemoveContainer" containerID="2f6a662cac7fed4d75b3ee7eca7be35e7cbf47c7b4f6472bf5e0bdf22ddf70b7"
Nov 02 15:06:40.275890 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:40.275872    5034 scope.go:122] "RemoveContainer" containerID="1829b69dcef0f61fb20aa350c19d9f3d6eba703895927a8fb2a3f08b9cb97469"
Nov 02 15:06:48.412424 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:06:48.412386    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:06:52.169284 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.169232    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz" podUID="deb89a97-000d-4a7f-80aa-9c4c00b45990" containerName="main" containerID="cri-o://0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac" gracePeriod=45
Nov 02 15:06:52.275616 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.275577    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz" podUID="deb89a97-000d-4a7f-80aa-9c4c00b45990" containerName="main" probeResult="failure" output="Get \"https://10.132.1.191:8000/ready\": dial tcp 10.132.1.191:8000: connect: connection refused"
Nov 02 15:06:52.428998 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.428942    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:52.538116 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.538078    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-home\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-home\") pod \"deb89a97-000d-4a7f-80aa-9c4c00b45990\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") "
Nov 02 15:06:52.538333 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.538176    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-model-cache\") pod \"deb89a97-000d-4a7f-80aa-9c4c00b45990\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") "
Nov 02 15:06:52.538333 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.538226    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb89a97-000d-4a7f-80aa-9c4c00b45990-kube-api-access-mh6l8\" (UniqueName: \"kubernetes.io/projected/deb89a97-000d-4a7f-80aa-9c4c00b45990-kube-api-access-mh6l8\") pod \"deb89a97-000d-4a7f-80aa-9c4c00b45990\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") "
Nov 02 15:06:52.538333 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.538271    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deb89a97-000d-4a7f-80aa-9c4c00b45990-tls-certs\" (UniqueName: \"kubernetes.io/secret/deb89a97-000d-4a7f-80aa-9c4c00b45990-tls-certs\") pod \"deb89a97-000d-4a7f-80aa-9c4c00b45990\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") "
Nov 02 15:06:52.538519 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.538355    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-dshm\") pod \"deb89a97-000d-4a7f-80aa-9c4c00b45990\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") "
Nov 02 15:06:52.538519 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.538455    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-tmp-dir\") pod \"deb89a97-000d-4a7f-80aa-9c4c00b45990\" (UID: \"deb89a97-000d-4a7f-80aa-9c4c00b45990\") "
Nov 02 15:06:52.539043 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.539009    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-tmp-dir" pod "deb89a97-000d-4a7f-80aa-9c4c00b45990" (UID: "deb89a97-000d-4a7f-80aa-9c4c00b45990"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:06:52.539214 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.539198    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-home" pod "deb89a97-000d-4a7f-80aa-9c4c00b45990" (UID: "deb89a97-000d-4a7f-80aa-9c4c00b45990"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:06:52.539377 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.539355    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-model-cache" pod "deb89a97-000d-4a7f-80aa-9c4c00b45990" (UID: "deb89a97-000d-4a7f-80aa-9c4c00b45990"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:06:52.550377 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.550312    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-dshm" pod "deb89a97-000d-4a7f-80aa-9c4c00b45990" (UID: "deb89a97-000d-4a7f-80aa-9c4c00b45990"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:06:52.551331 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.551241    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb89a97-000d-4a7f-80aa-9c4c00b45990-kube-api-access-mh6l8" pod "deb89a97-000d-4a7f-80aa-9c4c00b45990" (UID: "deb89a97-000d-4a7f-80aa-9c4c00b45990"). InnerVolumeSpecName "kube-api-access-mh6l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:06:52.553700 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.553652    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb89a97-000d-4a7f-80aa-9c4c00b45990-tls-certs" pod "deb89a97-000d-4a7f-80aa-9c4c00b45990" (UID: "deb89a97-000d-4a7f-80aa-9c4c00b45990"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.645517    5034 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-home\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.645558    5034 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-model-cache\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.645575    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh6l8\" (UniqueName: \"kubernetes.io/projected/deb89a97-000d-4a7f-80aa-9c4c00b45990-kube-api-access-mh6l8\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.645590    5034 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb89a97-000d-4a7f-80aa-9c4c00b45990-tls-certs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.645608    5034 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-dshm\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.645622    5034 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb89a97-000d-4a7f-80aa-9c4c00b45990-tmp-dir\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.648455    5034 generic.go:356] "Generic (PLEG): container finished" podID="deb89a97-000d-4a7f-80aa-9c4c00b45990" containerID="0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac" exitCode=0
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.648580    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz" event={"ID":"deb89a97-000d-4a7f-80aa-9c4c00b45990","Type":"ContainerDied","Data":"0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac"}
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.648617    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz" event={"ID":"deb89a97-000d-4a7f-80aa-9c4c00b45990","Type":"ContainerDied","Data":"59780f2bb5620a01151bb5a8b9a7d5b77bbcc54072ef519f0c4b5c22110ce09b"}
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.648643    5034 scope.go:122] "RemoveContainer" containerID="0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac"
Nov 02 15:06:52.649792 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.648835    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"
Nov 02 15:06:52.730801 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.728237    5034 scope.go:122] "RemoveContainer" containerID="0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac"
Nov 02 15:06:52.730801 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:06:52.729151    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac\": container with ID starting with 0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac not found: ID does not exist" containerID="0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac"
Nov 02 15:06:52.730801 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.729196    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac"} err="failed to get container status \"0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac\": rpc error: code = NotFound desc = could not find container \"0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac\": container with ID starting with 0c1fecbaa7ea94938c02fe25cf0521cd8485d6d7730a0b42382f6ca0393fe0ac not found: ID does not exist"
Nov 02 15:06:52.802802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.795288    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"]
Nov 02 15:06:52.815825 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:52.811180    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7540119b-kserve-68f5546f9c-r9vvz"]
Nov 02 15:06:53.426217 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:53.426182    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb89a97-000d-4a7f-80aa-9c4c00b45990" path="/var/lib/kubelet/pods/deb89a97-000d-4a7f-80aa-9c4c00b45990/volumes"
Nov 02 15:06:53.862037 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:53.861988    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:06:53.862405 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:53.862363    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:53.887905 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:53.887874    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:06:53.888138 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:53.888110    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:53.903025 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:53.902984    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:06:53.903238 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:53.903208    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:06:53.927989 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:53.927869    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:06:53.927989 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:06:53.927924    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:07:18.412702 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:07:18.412654    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:07:48.416948 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:07:48.416902    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:08:02.966555 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:02.966517    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-699c65dd57-f5zxl"]
Nov 02 15:08:02.967094 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:02.966960    5034 cpu_manager.go:414] "containerMap: removing container" podUID="deb89a97-000d-4a7f-80aa-9c4c00b45990" containerName="main"
Nov 02 15:08:02.967094 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:02.966973    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb89a97-000d-4a7f-80aa-9c4c00b45990" containerName="main"
Nov 02 15:08:02.967094 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:02.967073    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb89a97-000d-4a7f-80aa-9c4c00b45990" containerName="main"
Nov 02 15:08:02.970297 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:02.970276    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:02.990880 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:02.990848    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-699c65dd57-f5zxl"]
Nov 02 15:08:03.002136 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.002111    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 02 15:08:03.002284 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.002182    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-964jt\""
Nov 02 15:08:03.042824 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.042767    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0ae7b7-881d-4f6f-92df-7e5536bbeff9-plugins-config-volume\") pod \"payload-processing-699c65dd57-f5zxl\" (UID: \"4b0ae7b7-881d-4f6f-92df-7e5536bbeff9\") " pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:03.043006 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.042989    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4bcr\" (UniqueName: \"kubernetes.io/projected/4b0ae7b7-881d-4f6f-92df-7e5536bbeff9-kube-api-access-d4bcr\") pod \"payload-processing-699c65dd57-f5zxl\" (UID: \"4b0ae7b7-881d-4f6f-92df-7e5536bbeff9\") " pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:03.143609 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.143587    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0ae7b7-881d-4f6f-92df-7e5536bbeff9-plugins-config-volume\") pod \"payload-processing-699c65dd57-f5zxl\" (UID: \"4b0ae7b7-881d-4f6f-92df-7e5536bbeff9\") " pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:03.143729 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.143685    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4bcr\" (UniqueName: \"kubernetes.io/projected/4b0ae7b7-881d-4f6f-92df-7e5536bbeff9-kube-api-access-d4bcr\") pod \"payload-processing-699c65dd57-f5zxl\" (UID: \"4b0ae7b7-881d-4f6f-92df-7e5536bbeff9\") " pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:03.144337 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.144313    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b0ae7b7-881d-4f6f-92df-7e5536bbeff9-plugins-config-volume\") pod \"payload-processing-699c65dd57-f5zxl\" (UID: \"4b0ae7b7-881d-4f6f-92df-7e5536bbeff9\") " pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:03.171043 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.171010    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4bcr\" (UniqueName: \"kubernetes.io/projected/4b0ae7b7-881d-4f6f-92df-7e5536bbeff9-kube-api-access-d4bcr\") pod \"payload-processing-699c65dd57-f5zxl\" (UID: \"4b0ae7b7-881d-4f6f-92df-7e5536bbeff9\") " pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:03.283420 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.283393    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:03.581915 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:08:03.581885    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0ae7b7_881d_4f6f_92df_7e5536bbeff9.slice/crio-f38d2742f58d3cb0048d6c2eb1f4c4eddee73bc1da2e24815e5faf773e01f34e WatchSource:0}: Error finding container f38d2742f58d3cb0048d6c2eb1f4c4eddee73bc1da2e24815e5faf773e01f34e: Status 404 returned error can't find the container with id f38d2742f58d3cb0048d6c2eb1f4c4eddee73bc1da2e24815e5faf773e01f34e
Nov 02 15:08:03.582457 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:03.582432    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-699c65dd57-f5zxl"]
Nov 02 15:08:04.046425 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:04.046393    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-699c65dd57-f5zxl" event={"ID":"4b0ae7b7-881d-4f6f-92df-7e5536bbeff9","Type":"ContainerStarted","Data":"dcbcde5234865a44f7faf404c92b0eddaf8566dc99a1e126225a93a05dc01fc3"}
Nov 02 15:08:04.046848 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:04.046449    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-699c65dd57-f5zxl" event={"ID":"4b0ae7b7-881d-4f6f-92df-7e5536bbeff9","Type":"ContainerStarted","Data":"f38d2742f58d3cb0048d6c2eb1f4c4eddee73bc1da2e24815e5faf773e01f34e"}
Nov 02 15:08:04.046848 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:04.046491    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:04.090458 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:04.090408    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-699c65dd57-f5zxl" podStartSLOduration=2.090395862 podStartE2EDuration="2.090395862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:04.089942425 +0000 UTC m=+5074.841246865" watchObservedRunningTime="2026-11-02 15:08:04.090395862 +0000 UTC m=+5074.841700300"
Nov 02 15:08:15.054048 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:15.053962    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-699c65dd57-f5zxl"
Nov 02 15:08:18.414839 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:08:18.414800    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:08:31.112380 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:31.112354    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:08:31.132991 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:08:31.132971    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:08:48.413304 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:08:48.413261    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:09:18.412918 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:09:18.412876    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:09:23.039435 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:09:23.039400    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"]
Nov 02 15:09:23.053183 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:09:23.053155    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa65da05-f6tbx"]
Nov 02 15:09:23.392919 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:09:23.392847    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf61bf0f-3b68-46df-bf5e-507806ac85b2" path="/var/lib/kubelet/pods/bf61bf0f-3b68-46df-bf5e-507806ac85b2/volumes"
Nov 02 15:09:48.413258 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:09:48.413178    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:10:18.414287 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:10:18.414246    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:10:40.339641 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:10:40.339608    5034 scope.go:122] "RemoveContainer" containerID="0fb127da8e47affabe75a58bdfa7f704abd6d163b676d404e80c4d8424d55c20"
Nov 02 15:10:48.414378 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:10:48.414340    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:11:08.908005 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:11:08.907954    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:08.909262 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:11:08.909225    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:08.911958 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:11:08.911916    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:11:08.912193 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:11:08.912154    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:11:18.412515 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:11:18.412434    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:11:40.361414 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:11:40.361383    5034 scope.go:122] "RemoveContainer" containerID="d29c680c1a0c11b72606fb06df2a32c27c65a24b10f937b9810eeec171904031"
Nov 02 15:11:48.413311 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:11:48.413270    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:12:08.858803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:12:08.857866    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:08.858803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:12:08.857936    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:08.888800 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:12:08.888455    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:08.888800 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:12:08.888513    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:08.902801 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:12:08.900860    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:08.902801 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:12:08.900910    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:08.918889 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:12:08.918858    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:12:08.919211 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:12:08.919180    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:12:18.413120 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:12:18.413083    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:12:48.412713 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:12:48.412672    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:13:18.414368 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:13:18.414328    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:13:31.183218 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:13:31.183185    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:13:31.193023 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:13:31.192999    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:13:48.428346 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:13:48.428284    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:14:18.413117 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:14:18.413035    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:14:39.647755 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.647714    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"]
Nov 02 15:14:39.651974 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.651951    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:14:39.671032 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.670997    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"]
Nov 02 15:14:39.681939 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.681915    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-437128\""
Nov 02 15:14:39.715389 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.715361    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"]
Nov 02 15:14:39.718904 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.718884    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:39.736929 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.736903    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"]
Nov 02 15:14:39.737943 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.737921    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-49229c96-dockercfg-ks8hd\""
Nov 02 15:14:39.840130 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.840099    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7055def4-d7ac-4b0e-9490-594f4bb53a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj\" (UID: \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:39.840309 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.840150    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbmv\" (UniqueName: \"kubernetes.io/projected/7055def4-d7ac-4b0e-9490-594f4bb53a8d-kube-api-access-5bbmv\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj\" (UID: \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:39.840309 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.840207    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqlqv\" (UniqueName: \"kubernetes.io/projected/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-kube-api-access-bqlqv\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr\" (UID: \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:14:39.840427 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.840409    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr\" (UID: \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:14:39.941113 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.941036    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr\" (UID: \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:14:39.941113 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.941089    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7055def4-d7ac-4b0e-9490-594f4bb53a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj\" (UID: \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:39.941364 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.941121    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbmv\" (UniqueName: \"kubernetes.io/projected/7055def4-d7ac-4b0e-9490-594f4bb53a8d-kube-api-access-5bbmv\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj\" (UID: \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:39.941364 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.941146    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqlqv\" (UniqueName: \"kubernetes.io/projected/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-kube-api-access-bqlqv\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr\" (UID: \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:14:39.941894 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.941869    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr\" (UID: \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:14:39.942047 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.942025    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7055def4-d7ac-4b0e-9490-594f4bb53a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj\" (UID: \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:39.965228 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.965205    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqlqv\" (UniqueName: \"kubernetes.io/projected/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-kube-api-access-bqlqv\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr\" (UID: \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:14:39.965342 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.965322    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbmv\" (UniqueName: \"kubernetes.io/projected/7055def4-d7ac-4b0e-9490-594f4bb53a8d-kube-api-access-5bbmv\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj\" (UID: \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:39.965392 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:39.965339    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:14:40.033694 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.033656    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:40.284577 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.284221    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"]
Nov 02 15:14:40.349119 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.349093    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"]
Nov 02 15:14:40.349351 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:14:40.349330    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7055def4_d7ac_4b0e_9490_594f4bb53a8d.slice/crio-8636e237852b93316cc56a8cc814974f212b4aa584dfaa7de037198d7ad846b6 WatchSource:0}: Error finding container 8636e237852b93316cc56a8cc814974f212b4aa584dfaa7de037198d7ad846b6: Status 404 returned error can't find the container with id 8636e237852b93316cc56a8cc814974f212b4aa584dfaa7de037198d7ad846b6
Nov 02 15:14:40.624404 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.624321    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr" event={"ID":"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0","Type":"ContainerStarted","Data":"730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322"}
Nov 02 15:14:40.624404 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.624383    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr" event={"ID":"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0","Type":"ContainerStarted","Data":"3a4240bfc6f1fbf374f0c3951a511b39dc149d44c170b07a38e666ae2618fea3"}
Nov 02 15:14:40.624653 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.624529    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:14:40.626179 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.626149    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj" event={"ID":"7055def4-d7ac-4b0e-9490-594f4bb53a8d","Type":"ContainerStarted","Data":"a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc"}
Nov 02 15:14:40.626304 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.626201    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj" event={"ID":"7055def4-d7ac-4b0e-9490-594f4bb53a8d","Type":"ContainerStarted","Data":"8636e237852b93316cc56a8cc814974f212b4aa584dfaa7de037198d7ad846b6"}
Nov 02 15:14:40.626304 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.626229    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:40.667001 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.666948    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr" podStartSLOduration=1.66693259 podStartE2EDuration="1.66693259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:40.666128441 +0000 UTC m=+5471.417432880" watchObservedRunningTime="2026-11-02 15:14:40.66693259 +0000 UTC m=+5471.418237029"
Nov 02 15:14:40.708467 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:40.708414    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj" podStartSLOduration=1.708400303 podStartE2EDuration="1.708400303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:40.707526949 +0000 UTC m=+5471.458831411" watchObservedRunningTime="2026-11-02 15:14:40.708400303 +0000 UTC m=+5471.459704741"
Nov 02 15:14:48.414377 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:14:48.414261    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:14:51.634060 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:51.634030    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:14:51.634571 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:14:51.634096    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:15:00.150589 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.150551    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"]
Nov 02 15:15:00.154950 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.154925    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"]
Nov 02 15:15:00.155094 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.155041    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:00.158490 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.158472    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:00.171058 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.171037    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:15:00.171289 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.171275    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:15:00.184002 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.183975    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:15:00.186291 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.186271    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"]
Nov 02 15:15:00.188270 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.188220    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"]
Nov 02 15:15:00.251499 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.251468    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f34497-977a-4af1-9537-cdd5d2f60412-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-xjb5r\" (UID: \"d3f34497-977a-4af1-9537-cdd5d2f60412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:00.251645 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.251509    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84w2m\" (UniqueName: \"kubernetes.io/projected/d3f34497-977a-4af1-9537-cdd5d2f60412-kube-api-access-84w2m\") pod \"maas-api-key-cleanup-29893875-xjb5r\" (UID: \"d3f34497-977a-4af1-9537-cdd5d2f60412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:00.251645 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.251530    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws\" (UID: \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:00.251727 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.251677    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh4j2\" (UniqueName: \"kubernetes.io/projected/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-kube-api-access-mh4j2\") pod \"maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws\" (UID: \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:00.353118 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.353083    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4j2\" (UniqueName: \"kubernetes.io/projected/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-kube-api-access-mh4j2\") pod \"maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws\" (UID: \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:00.353348 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.353183    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f34497-977a-4af1-9537-cdd5d2f60412-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-xjb5r\" (UID: \"d3f34497-977a-4af1-9537-cdd5d2f60412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:00.353348 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.353229    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84w2m\" (UniqueName: \"kubernetes.io/projected/d3f34497-977a-4af1-9537-cdd5d2f60412-kube-api-access-84w2m\") pod \"maas-api-key-cleanup-29893875-xjb5r\" (UID: \"d3f34497-977a-4af1-9537-cdd5d2f60412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:00.353348 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.353251    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws\" (UID: \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:00.353883 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.353862    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f34497-977a-4af1-9537-cdd5d2f60412-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-xjb5r\" (UID: \"d3f34497-977a-4af1-9537-cdd5d2f60412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:00.353931 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.353889    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws\" (UID: \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:00.377529 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.377501    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh4j2\" (UniqueName: \"kubernetes.io/projected/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-kube-api-access-mh4j2\") pod \"maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws\" (UID: \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:00.377668 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.377553    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84w2m\" (UniqueName: \"kubernetes.io/projected/d3f34497-977a-4af1-9537-cdd5d2f60412-kube-api-access-84w2m\") pod \"maas-api-key-cleanup-29893875-xjb5r\" (UID: \"d3f34497-977a-4af1-9537-cdd5d2f60412\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:00.471346 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.471284    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:00.476306 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.476289    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:00.772932 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.772905    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"]
Nov 02 15:15:00.775037 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:15:00.775005    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5ea326_f5bd_4ef9_a5d0_eade95e17bb7.slice/crio-8ecd9a9d24827fba28219148bfde388f68a8876fa16974a6dcf3b597d31ec9f8 WatchSource:0}: Error finding container 8ecd9a9d24827fba28219148bfde388f68a8876fa16974a6dcf3b597d31ec9f8: Status 404 returned error can't find the container with id 8ecd9a9d24827fba28219148bfde388f68a8876fa16974a6dcf3b597d31ec9f8
Nov 02 15:15:00.789767 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:00.789735    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"]
Nov 02 15:15:00.791061 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:15:00.791035    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f34497_977a_4af1_9537_cdd5d2f60412.slice/crio-1aeb2df206b64282c18b37742ff3ce82a03f23301b590af7fea6a73f30c6d32c WatchSource:0}: Error finding container 1aeb2df206b64282c18b37742ff3ce82a03f23301b590af7fea6a73f30c6d32c: Status 404 returned error can't find the container with id 1aeb2df206b64282c18b37742ff3ce82a03f23301b590af7fea6a73f30c6d32c
Nov 02 15:15:01.741517 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:01.741477    5034 generic.go:356] "Generic (PLEG): container finished" podID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerID="8bc994ad28937c9a700f5fe98abc176f043037d98aa1b6066388507fea70b405" exitCode=60
Nov 02 15:15:01.741985 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:01.741560    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" event={"ID":"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7","Type":"ContainerDied","Data":"8bc994ad28937c9a700f5fe98abc176f043037d98aa1b6066388507fea70b405"}
Nov 02 15:15:01.741985 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:01.741618    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" event={"ID":"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7","Type":"ContainerStarted","Data":"8ecd9a9d24827fba28219148bfde388f68a8876fa16974a6dcf3b597d31ec9f8"}
Nov 02 15:15:01.741985 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:01.741875    5034 scope.go:122] "RemoveContainer" containerID="8bc994ad28937c9a700f5fe98abc176f043037d98aa1b6066388507fea70b405"
Nov 02 15:15:01.743470 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:01.743334    5034 generic.go:356] "Generic (PLEG): container finished" podID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerID="d76c9edbbc5b86750d83cb697c2ee0b29ba015030026d678035298b3c97f991a" exitCode=60
Nov 02 15:15:01.743470 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:01.743414    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" event={"ID":"d3f34497-977a-4af1-9537-cdd5d2f60412","Type":"ContainerDied","Data":"d76c9edbbc5b86750d83cb697c2ee0b29ba015030026d678035298b3c97f991a"}
Nov 02 15:15:01.743470 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:01.743456    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" event={"ID":"d3f34497-977a-4af1-9537-cdd5d2f60412","Type":"ContainerStarted","Data":"1aeb2df206b64282c18b37742ff3ce82a03f23301b590af7fea6a73f30c6d32c"}
Nov 02 15:15:01.743642 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:01.743597    5034 scope.go:122] "RemoveContainer" containerID="d76c9edbbc5b86750d83cb697c2ee0b29ba015030026d678035298b3c97f991a"
Nov 02 15:15:02.751118 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:02.751077    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" event={"ID":"d3f34497-977a-4af1-9537-cdd5d2f60412","Type":"ContainerStarted","Data":"215665c11ff053b9b33fe4df591c54dcdc9b1f5d8c4afc50847b0636af0ac2bb"}
Nov 02 15:15:02.753047 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:02.753018    5034 generic.go:356] "Generic (PLEG): container finished" podID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerID="7eb6f0cbc1b6008367251f7a9600ce42728b290a045262d62515144d1b0200b6" exitCode=60
Nov 02 15:15:02.753180 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:02.753056    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" event={"ID":"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7","Type":"ContainerDied","Data":"7eb6f0cbc1b6008367251f7a9600ce42728b290a045262d62515144d1b0200b6"}
Nov 02 15:15:02.753180 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:02.753096    5034 scope.go:122] "RemoveContainer" containerID="8bc994ad28937c9a700f5fe98abc176f043037d98aa1b6066388507fea70b405"
Nov 02 15:15:02.753341 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:02.753320    5034 scope.go:122] "RemoveContainer" containerID="7eb6f0cbc1b6008367251f7a9600ce42728b290a045262d62515144d1b0200b6"
Nov 02 15:15:02.753602 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:15:02.753581    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws_odh-ai-gateway-infra(3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7"
Nov 02 15:15:02.793440 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:02.793392    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" podStartSLOduration=2.793380544 podStartE2EDuration="2.793380544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:02.792711121 +0000 UTC m=+5493.544015561" watchObservedRunningTime="2026-11-02 15:15:02.793380544 +0000 UTC m=+5493.544684983"
Nov 02 15:15:03.760559 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:03.760525    5034 scope.go:122] "RemoveContainer" containerID="7eb6f0cbc1b6008367251f7a9600ce42728b290a045262d62515144d1b0200b6"
Nov 02 15:15:03.761067 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:15:03.760863    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws_odh-ai-gateway-infra(3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7"
Nov 02 15:15:03.762266 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:03.762238    5034 generic.go:356] "Generic (PLEG): container finished" podID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerID="215665c11ff053b9b33fe4df591c54dcdc9b1f5d8c4afc50847b0636af0ac2bb" exitCode=60
Nov 02 15:15:03.762381 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:03.762285    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" event={"ID":"d3f34497-977a-4af1-9537-cdd5d2f60412","Type":"ContainerDied","Data":"215665c11ff053b9b33fe4df591c54dcdc9b1f5d8c4afc50847b0636af0ac2bb"}
Nov 02 15:15:03.762381 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:03.762335    5034 scope.go:122] "RemoveContainer" containerID="d76c9edbbc5b86750d83cb697c2ee0b29ba015030026d678035298b3c97f991a"
Nov 02 15:15:03.762603 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:03.762583    5034 scope.go:122] "RemoveContainer" containerID="215665c11ff053b9b33fe4df591c54dcdc9b1f5d8c4afc50847b0636af0ac2bb"
Nov 02 15:15:03.762869 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:15:03.762845    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-xjb5r_odh-ai-gateway-infra(d3f34497-977a-4af1-9537-cdd5d2f60412)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412"
Nov 02 15:15:04.770069 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:04.770043    5034 scope.go:122] "RemoveContainer" containerID="215665c11ff053b9b33fe4df591c54dcdc9b1f5d8c4afc50847b0636af0ac2bb"
Nov 02 15:15:04.770501 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:15:04.770301    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-xjb5r_odh-ai-gateway-infra(d3f34497-977a-4af1-9537-cdd5d2f60412)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412"
Nov 02 15:15:14.387733 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:14.387697    5034 scope.go:122] "RemoveContainer" containerID="215665c11ff053b9b33fe4df591c54dcdc9b1f5d8c4afc50847b0636af0ac2bb"
Nov 02 15:15:14.838660 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:14.838622    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" event={"ID":"d3f34497-977a-4af1-9537-cdd5d2f60412","Type":"ContainerStarted","Data":"65232d104b765698a24c3e16089abe587c2068e4e20ecb4294bcd5c614eb748b"}
Nov 02 15:15:15.387526 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:15.387493    5034 scope.go:122] "RemoveContainer" containerID="7eb6f0cbc1b6008367251f7a9600ce42728b290a045262d62515144d1b0200b6"
Nov 02 15:15:15.845534 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:15.845499    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" event={"ID":"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7","Type":"ContainerStarted","Data":"5192fc1cbde2ee540391ec127f4867b9d0466803e66401d6f1c65efcb3257aa0"}
Nov 02 15:15:15.847360 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:15.847332    5034 generic.go:356] "Generic (PLEG): container finished" podID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerID="65232d104b765698a24c3e16089abe587c2068e4e20ecb4294bcd5c614eb748b" exitCode=60
Nov 02 15:15:15.847487 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:15.847405    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" event={"ID":"d3f34497-977a-4af1-9537-cdd5d2f60412","Type":"ContainerDied","Data":"65232d104b765698a24c3e16089abe587c2068e4e20ecb4294bcd5c614eb748b"}
Nov 02 15:15:15.847487 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:15.847459    5034 scope.go:122] "RemoveContainer" containerID="215665c11ff053b9b33fe4df591c54dcdc9b1f5d8c4afc50847b0636af0ac2bb"
Nov 02 15:15:15.847759 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:15.847742    5034 scope.go:122] "RemoveContainer" containerID="65232d104b765698a24c3e16089abe587c2068e4e20ecb4294bcd5c614eb748b"
Nov 02 15:15:15.848026 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:15:15.848003    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-xjb5r_odh-ai-gateway-infra(d3f34497-977a-4af1-9537-cdd5d2f60412)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412"
Nov 02 15:15:15.895455 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:15.895400    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" podStartSLOduration=15.895384785 podStartE2EDuration="15.895384785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:15.894177072 +0000 UTC m=+5506.645481511" watchObservedRunningTime="2026-11-02 15:15:15.895384785 +0000 UTC m=+5506.646689223"
Nov 02 15:15:15.899214 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:15.899181    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"]
Nov 02 15:15:16.854366 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:16.854334    5034 generic.go:356] "Generic (PLEG): container finished" podID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerID="5192fc1cbde2ee540391ec127f4867b9d0466803e66401d6f1c65efcb3257aa0" exitCode=60
Nov 02 15:15:16.854922 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:16.854410    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" event={"ID":"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7","Type":"ContainerDied","Data":"5192fc1cbde2ee540391ec127f4867b9d0466803e66401d6f1c65efcb3257aa0"}
Nov 02 15:15:16.854922 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:16.854477    5034 scope.go:122] "RemoveContainer" containerID="7eb6f0cbc1b6008367251f7a9600ce42728b290a045262d62515144d1b0200b6"
Nov 02 15:15:16.854922 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:16.854886    5034 scope.go:122] "RemoveContainer" containerID="5192fc1cbde2ee540391ec127f4867b9d0466803e66401d6f1c65efcb3257aa0"
Nov 02 15:15:16.855187 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:15:16.855164    5034 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws_odh-ai-gateway-infra(3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7"
Nov 02 15:15:16.906064 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:16.906032    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"]
Nov 02 15:15:17.014634 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.014608    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:17.139121 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.139024    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f34497-977a-4af1-9537-cdd5d2f60412-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f34497-977a-4af1-9537-cdd5d2f60412-maas-api-service-ca\") pod \"d3f34497-977a-4af1-9537-cdd5d2f60412\" (UID: \"d3f34497-977a-4af1-9537-cdd5d2f60412\") "
Nov 02 15:15:17.139319 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.139128    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f34497-977a-4af1-9537-cdd5d2f60412-kube-api-access-84w2m\" (UniqueName: \"kubernetes.io/projected/d3f34497-977a-4af1-9537-cdd5d2f60412-kube-api-access-84w2m\") pod \"d3f34497-977a-4af1-9537-cdd5d2f60412\" (UID: \"d3f34497-977a-4af1-9537-cdd5d2f60412\") "
Nov 02 15:15:17.139435 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.139407    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f34497-977a-4af1-9537-cdd5d2f60412-maas-api-service-ca" pod "d3f34497-977a-4af1-9537-cdd5d2f60412" (UID: "d3f34497-977a-4af1-9537-cdd5d2f60412"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:17.141438 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.141413    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f34497-977a-4af1-9537-cdd5d2f60412-kube-api-access-84w2m" pod "d3f34497-977a-4af1-9537-cdd5d2f60412" (UID: "d3f34497-977a-4af1-9537-cdd5d2f60412"). InnerVolumeSpecName "kube-api-access-84w2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:17.240448 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.240401    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f34497-977a-4af1-9537-cdd5d2f60412-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:15:17.240448 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.240438    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84w2m\" (UniqueName: \"kubernetes.io/projected/d3f34497-977a-4af1-9537-cdd5d2f60412-kube-api-access-84w2m\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:15:17.862632 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.862602    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"
Nov 02 15:15:17.863194 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.862636    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r" event={"ID":"d3f34497-977a-4af1-9537-cdd5d2f60412","Type":"ContainerDied","Data":"1aeb2df206b64282c18b37742ff3ce82a03f23301b590af7fea6a73f30c6d32c"}
Nov 02 15:15:17.863194 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.862698    5034 scope.go:122] "RemoveContainer" containerID="65232d104b765698a24c3e16089abe587c2068e4e20ecb4294bcd5c614eb748b"
Nov 02 15:15:17.916738 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.916702    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"]
Nov 02 15:15:17.929922 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:17.929887    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-xjb5r"]
Nov 02 15:15:18.017884 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.017857    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:18.151436 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.151351    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-kube-api-access-mh4j2\" (UniqueName: \"kubernetes.io/projected/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-kube-api-access-mh4j2\") pod \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\" (UID: \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\") "
Nov 02 15:15:18.151595 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.151452    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-maas-api-service-ca\") pod \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\" (UID: \"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7\") "
Nov 02 15:15:18.151838 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.151816    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-maas-api-service-ca" pod "3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" (UID: "3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:18.153577 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.153557    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-kube-api-access-mh4j2" pod "3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" (UID: "3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7"). InnerVolumeSpecName "kube-api-access-mh4j2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:18.252918 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.252873    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh4j2\" (UniqueName: \"kubernetes.io/projected/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-kube-api-access-mh4j2\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:15:18.252918 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.252909    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7-maas-api-service-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:15:18.422861 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:15:18.422740    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:15:18.871065 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.871032    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"
Nov 02 15:15:18.871065 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.871048    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws" event={"ID":"3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7","Type":"ContainerDied","Data":"8ecd9a9d24827fba28219148bfde388f68a8876fa16974a6dcf3b597d31ec9f8"}
Nov 02 15:15:18.871684 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.871113    5034 scope.go:122] "RemoveContainer" containerID="5192fc1cbde2ee540391ec127f4867b9d0466803e66401d6f1c65efcb3257aa0"
Nov 02 15:15:18.924030 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.923991    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"]
Nov 02 15:15:18.927847 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:18.927816    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-437128-2989387jhxws"]
Nov 02 15:15:19.393235 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:19.393199    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" path="/var/lib/kubelet/pods/3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7/volumes"
Nov 02 15:15:19.393718 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:19.393697    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" path="/var/lib/kubelet/pods/d3f34497-977a-4af1-9537-cdd5d2f60412/volumes"
Nov 02 15:15:27.611385 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611348    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"]
Nov 02 15:15:27.611852 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611837    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerName="cleanup"
Nov 02 15:15:27.611905 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611853    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerName="cleanup"
Nov 02 15:15:27.611905 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611869    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerName="cleanup"
Nov 02 15:15:27.611905 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611877    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerName="cleanup"
Nov 02 15:15:27.611905 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611889    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerName="cleanup"
Nov 02 15:15:27.611905 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611895    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerName="cleanup"
Nov 02 15:15:27.612082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611913    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerName="cleanup"
Nov 02 15:15:27.612082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611917    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerName="cleanup"
Nov 02 15:15:27.612082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.611998    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerName="cleanup"
Nov 02 15:15:27.612082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.612007    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerName="cleanup"
Nov 02 15:15:27.612082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.612014    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerName="cleanup"
Nov 02 15:15:27.612082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.612022    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerName="cleanup"
Nov 02 15:15:27.612082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.612028    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerName="cleanup"
Nov 02 15:15:27.612082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.612034    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerName="cleanup"
Nov 02 15:15:27.616423 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.616404    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.632545 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.632512    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:15:27.634341 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.634307    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"]
Nov 02 15:15:27.645137 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.645099    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2ebb5d-5z2d6\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.645339 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.645157    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2ebb5d-5z2d6\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.645339 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.645265    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2hm9\" (UniqueName: \"kubernetes.io/projected/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-kube-api-access-p2hm9\") pod \"maas-api-revoke-sub-9e2ebb5d-5z2d6\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.645339 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.645305    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:15:27.645339 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.645309    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:15:27.645599 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.645365    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:15:27.746704 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.746669    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2ebb5d-5z2d6\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.746929 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.746716    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2ebb5d-5z2d6\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.746929 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.746839    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2hm9\" (UniqueName: \"kubernetes.io/projected/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-kube-api-access-p2hm9\") pod \"maas-api-revoke-sub-9e2ebb5d-5z2d6\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.747408 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.747387    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e2ebb5d-5z2d6\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.749062 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.749036    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e2ebb5d-5z2d6\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.771470 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.771445    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2hm9\" (UniqueName: \"kubernetes.io/projected/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-kube-api-access-p2hm9\") pod \"maas-api-revoke-sub-9e2ebb5d-5z2d6\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:27.929394 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:27.929313    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:28.228310 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:28.228281    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"]
Nov 02 15:15:28.228494 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:15:28.228467    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a35f36_8bab_4fd4_aa74_4cb2073b92c7.slice/crio-f76719e289d7bb57b02c4afed0d4dab010f10590ab1d0289ddcee23f2e82aa87 WatchSource:0}: Error finding container f76719e289d7bb57b02c4afed0d4dab010f10590ab1d0289ddcee23f2e82aa87: Status 404 returned error can't find the container with id f76719e289d7bb57b02c4afed0d4dab010f10590ab1d0289ddcee23f2e82aa87
Nov 02 15:15:28.937806 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:28.937746    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6" event={"ID":"26a35f36-8bab-4fd4-aa74-4cb2073b92c7","Type":"ContainerStarted","Data":"5a98b23f21dbd4904a6335bf41576d892475deaa84bba44752eb4e4ce7324155"}
Nov 02 15:15:28.938284 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:28.937868    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6" event={"ID":"26a35f36-8bab-4fd4-aa74-4cb2073b92c7","Type":"ContainerStarted","Data":"f76719e289d7bb57b02c4afed0d4dab010f10590ab1d0289ddcee23f2e82aa87"}
Nov 02 15:15:28.980623 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:28.980572    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6" podStartSLOduration=1.980559658 podStartE2EDuration="1.980559658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:28.979233671 +0000 UTC m=+5519.730538109" watchObservedRunningTime="2026-11-02 15:15:28.980559658 +0000 UTC m=+5519.731864116"
Nov 02 15:15:29.944578 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:29.944542    5034 generic.go:356] "Generic (PLEG): container finished" podID="26a35f36-8bab-4fd4-aa74-4cb2073b92c7" containerID="5a98b23f21dbd4904a6335bf41576d892475deaa84bba44752eb4e4ce7324155" exitCode=0
Nov 02 15:15:29.945023 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:29.944624    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6" event={"ID":"26a35f36-8bab-4fd4-aa74-4cb2073b92c7","Type":"ContainerDied","Data":"5a98b23f21dbd4904a6335bf41576d892475deaa84bba44752eb4e4ce7324155"}
Nov 02 15:15:31.097235 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.097205    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:31.176980 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.176950    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-xks-ca\") pod \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") "
Nov 02 15:15:31.177179 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.176989    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-openshift-ca\") pod \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") "
Nov 02 15:15:31.177179 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.177024    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-kube-api-access-p2hm9\" (UniqueName: \"kubernetes.io/projected/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-kube-api-access-p2hm9\") pod \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\" (UID: \"26a35f36-8bab-4fd4-aa74-4cb2073b92c7\") "
Nov 02 15:15:31.177437 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.177412    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-openshift-ca" pod "26a35f36-8bab-4fd4-aa74-4cb2073b92c7" (UID: "26a35f36-8bab-4fd4-aa74-4cb2073b92c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:15:31.179250 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.179226    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-kube-api-access-p2hm9" pod "26a35f36-8bab-4fd4-aa74-4cb2073b92c7" (UID: "26a35f36-8bab-4fd4-aa74-4cb2073b92c7"). InnerVolumeSpecName "kube-api-access-p2hm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:15:31.179343 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.179229    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-xks-ca" pod "26a35f36-8bab-4fd4-aa74-4cb2073b92c7" (UID: "26a35f36-8bab-4fd4-aa74-4cb2073b92c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:15:31.278482 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.278459    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:15:31.278596 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.278484    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:15:31.278596 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.278495    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2hm9\" (UniqueName: \"kubernetes.io/projected/26a35f36-8bab-4fd4-aa74-4cb2073b92c7-kube-api-access-p2hm9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:15:31.956721 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.956691    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"
Nov 02 15:15:31.956721 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.956699    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6" event={"ID":"26a35f36-8bab-4fd4-aa74-4cb2073b92c7","Type":"ContainerDied","Data":"f76719e289d7bb57b02c4afed0d4dab010f10590ab1d0289ddcee23f2e82aa87"}
Nov 02 15:15:31.956961 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:31.956743    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f76719e289d7bb57b02c4afed0d4dab010f10590ab1d0289ddcee23f2e82aa87"
Nov 02 15:15:44.621202 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:44.621155    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"]
Nov 02 15:15:44.652283 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:44.652245    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"]
Nov 02 15:15:48.414510 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:15:48.414472    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:15:49.621727 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:49.621658    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr" podUID="8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" containerName="payload-processing" containerID="cri-o://730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322" gracePeriod=25
Nov 02 15:15:49.653303 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:49.653252    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj" podUID="7055def4-d7ac-4b0e-9490-594f4bb53a8d" containerName="payload-pre-processing" containerID="cri-o://a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc" gracePeriod=25
Nov 02 15:15:51.632995 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:51.632954    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj" podUID="7055def4-d7ac-4b0e-9490-594f4bb53a8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:15:51.633463 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:15:51.632960    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr" podUID="8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:16:01.632321 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:01.632284    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr" podUID="8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:16:01.632748 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:01.632282    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj" podUID="7055def4-d7ac-4b0e-9490-594f4bb53a8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 15:16:09.913984 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:09.913958    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:16:09.946047 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:09.946026    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:16:10.058451 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.058415    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-plugins-config-volume\") pod \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\" (UID: \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\") "
Nov 02 15:16:10.058664 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.058493    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7055def4-d7ac-4b0e-9490-594f4bb53a8d-kube-api-access-5bbmv\" (UniqueName: \"kubernetes.io/projected/7055def4-d7ac-4b0e-9490-594f4bb53a8d-kube-api-access-5bbmv\") pod \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\" (UID: \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\") "
Nov 02 15:16:10.058664 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.058526    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-kube-api-access-bqlqv\" (UniqueName: \"kubernetes.io/projected/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-kube-api-access-bqlqv\") pod \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\" (UID: \"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0\") "
Nov 02 15:16:10.058664 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.058563    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7055def4-d7ac-4b0e-9490-594f4bb53a8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7055def4-d7ac-4b0e-9490-594f4bb53a8d-plugins-config-volume\") pod \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\" (UID: \"7055def4-d7ac-4b0e-9490-594f4bb53a8d\") "
Nov 02 15:16:10.059069 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.059035    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-plugins-config-volume" pod "8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" (UID: "8fa9d853-09cf-4d90-a21c-aae09e1d2ac0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:10.059192 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.059067    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7055def4-d7ac-4b0e-9490-594f4bb53a8d-plugins-config-volume" pod "7055def4-d7ac-4b0e-9490-594f4bb53a8d" (UID: "7055def4-d7ac-4b0e-9490-594f4bb53a8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:10.060759 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.060730    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-kube-api-access-bqlqv" pod "8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" (UID: "8fa9d853-09cf-4d90-a21c-aae09e1d2ac0"). InnerVolumeSpecName "kube-api-access-bqlqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:10.060759 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.060746    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7055def4-d7ac-4b0e-9490-594f4bb53a8d-kube-api-access-5bbmv" pod "7055def4-d7ac-4b0e-9490-594f4bb53a8d" (UID: "7055def4-d7ac-4b0e-9490-594f4bb53a8d"). InnerVolumeSpecName "kube-api-access-5bbmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:10.153508 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.153442    5034 generic.go:356] "Generic (PLEG): container finished" podID="7055def4-d7ac-4b0e-9490-594f4bb53a8d" containerID="a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc" exitCode=0
Nov 02 15:16:10.153657 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.153516    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"
Nov 02 15:16:10.153657 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.153533    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj" event={"ID":"7055def4-d7ac-4b0e-9490-594f4bb53a8d","Type":"ContainerDied","Data":"a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc"}
Nov 02 15:16:10.153657 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.153623    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj" event={"ID":"7055def4-d7ac-4b0e-9490-594f4bb53a8d","Type":"ContainerDied","Data":"8636e237852b93316cc56a8cc814974f212b4aa584dfaa7de037198d7ad846b6"}
Nov 02 15:16:10.153657 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.153655    5034 scope.go:122] "RemoveContainer" containerID="a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc"
Nov 02 15:16:10.154992 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.154970    5034 generic.go:356] "Generic (PLEG): container finished" podID="8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" containerID="730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322" exitCode=0
Nov 02 15:16:10.155070 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.155026    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"
Nov 02 15:16:10.155070 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.155043    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr" event={"ID":"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0","Type":"ContainerDied","Data":"730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322"}
Nov 02 15:16:10.155176 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.155082    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr" event={"ID":"8fa9d853-09cf-4d90-a21c-aae09e1d2ac0","Type":"ContainerDied","Data":"3a4240bfc6f1fbf374f0c3951a511b39dc149d44c170b07a38e666ae2618fea3"}
Nov 02 15:16:10.159319 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.159294    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbmv\" (UniqueName: \"kubernetes.io/projected/7055def4-d7ac-4b0e-9490-594f4bb53a8d-kube-api-access-5bbmv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:10.159433 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.159324    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqlqv\" (UniqueName: \"kubernetes.io/projected/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-kube-api-access-bqlqv\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:10.159433 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.159339    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7055def4-d7ac-4b0e-9490-594f4bb53a8d-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:10.159433 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.159352    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:10.164475 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.164454    5034 scope.go:122] "RemoveContainer" containerID="a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc"
Nov 02 15:16:10.164764 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:16:10.164737    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc\": container with ID starting with a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc not found: ID does not exist" containerID="a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc"
Nov 02 15:16:10.164903 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.164787    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc"} err="failed to get container status \"a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc\": rpc error: code = NotFound desc = could not find container \"a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc\": container with ID starting with a68d9add4503cc1d8e0405a8adeb299e5f53ad01086f084384029aca93d6eecc not found: ID does not exist"
Nov 02 15:16:10.164903 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.164811    5034 scope.go:122] "RemoveContainer" containerID="730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322"
Nov 02 15:16:10.173186 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.173169    5034 scope.go:122] "RemoveContainer" containerID="730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322"
Nov 02 15:16:10.173439 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:16:10.173421    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322\": container with ID starting with 730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322 not found: ID does not exist" containerID="730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322"
Nov 02 15:16:10.173501 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.173445    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322"} err="failed to get container status \"730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322\": rpc error: code = NotFound desc = could not find container \"730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322\": container with ID starting with 730e0670b45d5f08829fea758cc6e2e732ff8ef07cfcc6904600c0789b598322 not found: ID does not exist"
Nov 02 15:16:10.208001 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.207982    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"]
Nov 02 15:16:10.211845 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.211815    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f95bznrr"]
Nov 02 15:16:10.247985 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.247961    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"]
Nov 02 15:16:10.252524 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:10.252505    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dx4pvj"]
Nov 02 15:16:11.393104 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:11.393074    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7055def4-d7ac-4b0e-9490-594f4bb53a8d" path="/var/lib/kubelet/pods/7055def4-d7ac-4b0e-9490-594f4bb53a8d/volumes"
Nov 02 15:16:11.393492 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:11.393437    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" path="/var/lib/kubelet/pods/8fa9d853-09cf-4d90-a21c-aae09e1d2ac0/volumes"
Nov 02 15:16:18.412914 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:16:18.412876    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:16:19.733821 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.733766    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"]
Nov 02 15:16:19.734253 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734237    5034 cpu_manager.go:414] "containerMap: removing container" podUID="26a35f36-8bab-4fd4-aa74-4cb2073b92c7" containerName="revoke-keys"
Nov 02 15:16:19.734310 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734254    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26a35f36-8bab-4fd4-aa74-4cb2073b92c7" containerName="revoke-keys"
Nov 02 15:16:19.734310 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734276    5034 cpu_manager.go:414] "containerMap: removing container" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerName="cleanup"
Nov 02 15:16:19.734310 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734280    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5ea326-f5bd-4ef9-a5d0-eade95e17bb7" containerName="cleanup"
Nov 02 15:16:19.734310 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734296    5034 cpu_manager.go:414] "containerMap: removing container" podUID="8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" containerName="payload-processing"
Nov 02 15:16:19.734310 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734303    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" containerName="payload-processing"
Nov 02 15:16:19.734310 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734309    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7055def4-d7ac-4b0e-9490-594f4bb53a8d" containerName="payload-pre-processing"
Nov 02 15:16:19.734523 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734314    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7055def4-d7ac-4b0e-9490-594f4bb53a8d" containerName="payload-pre-processing"
Nov 02 15:16:19.734523 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734324    5034 cpu_manager.go:414] "containerMap: removing container" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerName="cleanup"
Nov 02 15:16:19.734523 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734330    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f34497-977a-4af1-9537-cdd5d2f60412" containerName="cleanup"
Nov 02 15:16:19.734523 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734391    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fa9d853-09cf-4d90-a21c-aae09e1d2ac0" containerName="payload-processing"
Nov 02 15:16:19.734523 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734399    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7055def4-d7ac-4b0e-9490-594f4bb53a8d" containerName="payload-pre-processing"
Nov 02 15:16:19.734523 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.734411    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="26a35f36-8bab-4fd4-aa74-4cb2073b92c7" containerName="revoke-keys"
Nov 02 15:16:19.737460 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.737445    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:19.762943 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.762906    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"]
Nov 02 15:16:19.773323 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.773291    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-437128\""
Nov 02 15:16:19.809101 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.809060    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"]
Nov 02 15:16:19.812744 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.812720    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:19.830712 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.830683    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-49229c96-dockercfg-2m74f\""
Nov 02 15:16:19.836133 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.836105    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"]
Nov 02 15:16:19.847246 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.847213    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb8ll\" (UniqueName: \"kubernetes.io/projected/82b074ff-015b-4d8b-92ec-60d3cc4e2778-kube-api-access-cb8ll\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f952v58b\" (UID: \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:19.847412 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.847274    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fctc\" (UniqueName: \"kubernetes.io/projected/08bf8963-4514-481e-99d1-c2cf3eaf88cb-kube-api-access-9fctc\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5\" (UID: \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:19.847491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.847443    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bf8963-4514-481e-99d1-c2cf3eaf88cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5\" (UID: \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:19.847557 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.847499    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b074ff-015b-4d8b-92ec-60d3cc4e2778-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f952v58b\" (UID: \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:19.966802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.955533    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb8ll\" (UniqueName: \"kubernetes.io/projected/82b074ff-015b-4d8b-92ec-60d3cc4e2778-kube-api-access-cb8ll\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f952v58b\" (UID: \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:19.966802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.955614    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fctc\" (UniqueName: \"kubernetes.io/projected/08bf8963-4514-481e-99d1-c2cf3eaf88cb-kube-api-access-9fctc\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5\" (UID: \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:19.966802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.955691    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bf8963-4514-481e-99d1-c2cf3eaf88cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5\" (UID: \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:19.966802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.955730    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b074ff-015b-4d8b-92ec-60d3cc4e2778-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f952v58b\" (UID: \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:19.966802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.956669    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b074ff-015b-4d8b-92ec-60d3cc4e2778-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f952v58b\" (UID: \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:19.966802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.956928    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bf8963-4514-481e-99d1-c2cf3eaf88cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5\" (UID: \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:19.985957 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.982173    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb8ll\" (UniqueName: \"kubernetes.io/projected/82b074ff-015b-4d8b-92ec-60d3cc4e2778-kube-api-access-cb8ll\") pod \"payload-processing-e2e-worker-single-main-437128-7fd447f952v58b\" (UID: \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:19.985957 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:19.983277    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fctc\" (UniqueName: \"kubernetes.io/projected/08bf8963-4514-481e-99d1-c2cf3eaf88cb-kube-api-access-9fctc\") pod \"payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5\" (UID: \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:20.051999 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:20.051956    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:20.128773 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:20.128742    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:20.361048 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:16:20.361016    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b074ff_015b_4d8b_92ec_60d3cc4e2778.slice/crio-5333ad5558111b96b88a6fb4a4467efcb82561921717b5b21b8efd64bb907de2 WatchSource:0}: Error finding container 5333ad5558111b96b88a6fb4a4467efcb82561921717b5b21b8efd64bb907de2: Status 404 returned error can't find the container with id 5333ad5558111b96b88a6fb4a4467efcb82561921717b5b21b8efd64bb907de2
Nov 02 15:16:20.361188 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:20.361066    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"]
Nov 02 15:16:20.471369 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:20.471342    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"]
Nov 02 15:16:20.471482 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:16:20.471344    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bf8963_4514_481e_99d1_c2cf3eaf88cb.slice/crio-daa418927c8f4a5ae9cf559b2d66c9722902f904b19f851111db4cdddfdfb4f5 WatchSource:0}: Error finding container daa418927c8f4a5ae9cf559b2d66c9722902f904b19f851111db4cdddfdfb4f5: Status 404 returned error can't find the container with id daa418927c8f4a5ae9cf559b2d66c9722902f904b19f851111db4cdddfdfb4f5
Nov 02 15:16:21.219660 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:21.219607    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b" event={"ID":"82b074ff-015b-4d8b-92ec-60d3cc4e2778","Type":"ContainerStarted","Data":"5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87"}
Nov 02 15:16:21.220158 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:21.219680    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b" event={"ID":"82b074ff-015b-4d8b-92ec-60d3cc4e2778","Type":"ContainerStarted","Data":"5333ad5558111b96b88a6fb4a4467efcb82561921717b5b21b8efd64bb907de2"}
Nov 02 15:16:21.220158 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:21.219806    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:21.221392 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:21.221362    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5" event={"ID":"08bf8963-4514-481e-99d1-c2cf3eaf88cb","Type":"ContainerStarted","Data":"c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79"}
Nov 02 15:16:21.221540 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:21.221407    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5" event={"ID":"08bf8963-4514-481e-99d1-c2cf3eaf88cb","Type":"ContainerStarted","Data":"daa418927c8f4a5ae9cf559b2d66c9722902f904b19f851111db4cdddfdfb4f5"}
Nov 02 15:16:21.221540 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:21.221456    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:21.266487 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:21.266430    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b" podStartSLOduration=2.266411636 podStartE2EDuration="2.266411636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:21.264615993 +0000 UTC m=+5572.015920431" watchObservedRunningTime="2026-11-02 15:16:21.266411636 +0000 UTC m=+5572.017716075"
Nov 02 15:16:21.309928 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:21.309866    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5" podStartSLOduration=2.309846772 podStartE2EDuration="2.309846772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:21.30636112 +0000 UTC m=+5572.057665580" watchObservedRunningTime="2026-11-02 15:16:21.309846772 +0000 UTC m=+5572.061151212"
Nov 02 15:16:29.880561 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.880523    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"]
Nov 02 15:16:29.884825 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.884799    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:29.914865 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.906461    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:16:29.914865 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.906609    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:16:29.914865 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.906691    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:16:29.921150 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.921121    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:16:29.924717 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.924684    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"]
Nov 02 15:16:29.972891 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.972508    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:29.972891 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.972570    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:29.972891 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:29.972632    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6b9\" (UniqueName: \"kubernetes.io/projected/f81fb52d-c852-49c8-b8cc-2ccc087ff260-kube-api-access-tl6b9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:30.073527 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.073495    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:30.073734 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.073555    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6b9\" (UniqueName: \"kubernetes.io/projected/f81fb52d-c852-49c8-b8cc-2ccc087ff260-kube-api-access-tl6b9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:30.073734 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.073669    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:30.075944 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.075916    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:30.088332 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.088301    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:16:30.094748 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.094727    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:30.110061 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.110041    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:16:30.118339 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.118317    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6b9\" (UniqueName: \"kubernetes.io/projected/f81fb52d-c852-49c8-b8cc-2ccc087ff260-kube-api-access-tl6b9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:30.228487 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.228412    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:30.529252 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:30.529224    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"]
Nov 02 15:16:30.529414 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:16:30.529315    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81fb52d_c852_49c8_b8cc_2ccc087ff260.slice/crio-3e692c407615cc44151e93185eb8c51da8cb46f76c8c7ef7dc18487dbfc8072e WatchSource:0}: Error finding container 3e692c407615cc44151e93185eb8c51da8cb46f76c8c7ef7dc18487dbfc8072e: Status 404 returned error can't find the container with id 3e692c407615cc44151e93185eb8c51da8cb46f76c8c7ef7dc18487dbfc8072e
Nov 02 15:16:31.280188 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:31.280083    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc" event={"ID":"f81fb52d-c852-49c8-b8cc-2ccc087ff260","Type":"ContainerStarted","Data":"5cb140948d393ad6b6552101d61f638f8146776bf9f79b84dfca8b2c9ca83fe8"}
Nov 02 15:16:31.280188 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:31.280140    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc" event={"ID":"f81fb52d-c852-49c8-b8cc-2ccc087ff260","Type":"ContainerStarted","Data":"3e692c407615cc44151e93185eb8c51da8cb46f76c8c7ef7dc18487dbfc8072e"}
Nov 02 15:16:32.229974 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.229947    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:32.230244 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.230020    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:32.287807 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.287754    5034 generic.go:356] "Generic (PLEG): container finished" podID="f81fb52d-c852-49c8-b8cc-2ccc087ff260" containerID="5cb140948d393ad6b6552101d61f638f8146776bf9f79b84dfca8b2c9ca83fe8" exitCode=0
Nov 02 15:16:32.288217 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.287815    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc" event={"ID":"f81fb52d-c852-49c8-b8cc-2ccc087ff260","Type":"ContainerDied","Data":"5cb140948d393ad6b6552101d61f638f8146776bf9f79b84dfca8b2c9ca83fe8"}
Nov 02 15:16:32.449385 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.449363    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:32.597649 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.597623    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-openshift-ca\") pod \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") "
Nov 02 15:16:32.597839 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.597683    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81fb52d-c852-49c8-b8cc-2ccc087ff260-kube-api-access-tl6b9\" (UniqueName: \"kubernetes.io/projected/f81fb52d-c852-49c8-b8cc-2ccc087ff260-kube-api-access-tl6b9\") pod \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") "
Nov 02 15:16:32.597839 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.597809    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-xks-ca\") pod \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\" (UID: \"f81fb52d-c852-49c8-b8cc-2ccc087ff260\") "
Nov 02 15:16:32.598051 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.598032    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-openshift-ca" pod "f81fb52d-c852-49c8-b8cc-2ccc087ff260" (UID: "f81fb52d-c852-49c8-b8cc-2ccc087ff260"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:32.598181 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.598165    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:32.599651 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.599625    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-xks-ca" pod "f81fb52d-c852-49c8-b8cc-2ccc087ff260" (UID: "f81fb52d-c852-49c8-b8cc-2ccc087ff260"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:16:32.599765 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.599748    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81fb52d-c852-49c8-b8cc-2ccc087ff260-kube-api-access-tl6b9" pod "f81fb52d-c852-49c8-b8cc-2ccc087ff260" (UID: "f81fb52d-c852-49c8-b8cc-2ccc087ff260"). InnerVolumeSpecName "kube-api-access-tl6b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:32.699008 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.698979    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6b9\" (UniqueName: \"kubernetes.io/projected/f81fb52d-c852-49c8-b8cc-2ccc087ff260-kube-api-access-tl6b9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:32.699008 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:32.699005    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f81fb52d-c852-49c8-b8cc-2ccc087ff260-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:33.294316 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:33.294279    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc" event={"ID":"f81fb52d-c852-49c8-b8cc-2ccc087ff260","Type":"ContainerDied","Data":"3e692c407615cc44151e93185eb8c51da8cb46f76c8c7ef7dc18487dbfc8072e"}
Nov 02 15:16:33.294731 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:33.294342    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e692c407615cc44151e93185eb8c51da8cb46f76c8c7ef7dc18487dbfc8072e"
Nov 02 15:16:33.294731 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:33.294290    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"
Nov 02 15:16:40.300317 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:40.300280    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"]
Nov 02 15:16:40.334949 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:40.334918    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"]
Nov 02 15:16:43.524120 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:43.524086    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"]
Nov 02 15:16:43.537000 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:43.536970    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-437128-h4gqc"]
Nov 02 15:16:45.301294 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.301245    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b" podUID="82b074ff-015b-4d8b-92ec-60d3cc4e2778" containerName="payload-processing" containerID="cri-o://5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87" gracePeriod=25
Nov 02 15:16:45.336117 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.336060    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5" podUID="08bf8963-4514-481e-99d1-c2cf3eaf88cb" containerName="payload-pre-processing" containerID="cri-o://c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79" gracePeriod=25
Nov 02 15:16:45.393258 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.393232    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f81fb52d-c852-49c8-b8cc-2ccc087ff260" path="/var/lib/kubelet/pods/f81fb52d-c852-49c8-b8cc-2ccc087ff260/volumes"
Nov 02 15:16:45.596255 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.596227    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:45.617140 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.617037    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b074ff-015b-4d8b-92ec-60d3cc4e2778-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b074ff-015b-4d8b-92ec-60d3cc4e2778-plugins-config-volume\") pod \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\" (UID: \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\") "
Nov 02 15:16:45.617140 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.617103    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b074ff-015b-4d8b-92ec-60d3cc4e2778-kube-api-access-cb8ll\" (UniqueName: \"kubernetes.io/projected/82b074ff-015b-4d8b-92ec-60d3cc4e2778-kube-api-access-cb8ll\") pod \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\" (UID: \"82b074ff-015b-4d8b-92ec-60d3cc4e2778\") "
Nov 02 15:16:45.617545 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.617444    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b074ff-015b-4d8b-92ec-60d3cc4e2778-plugins-config-volume" pod "82b074ff-015b-4d8b-92ec-60d3cc4e2778" (UID: "82b074ff-015b-4d8b-92ec-60d3cc4e2778"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:45.619376 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.619348    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b074ff-015b-4d8b-92ec-60d3cc4e2778-kube-api-access-cb8ll" pod "82b074ff-015b-4d8b-92ec-60d3cc4e2778" (UID: "82b074ff-015b-4d8b-92ec-60d3cc4e2778"). InnerVolumeSpecName "kube-api-access-cb8ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:45.635451 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.635430    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:45.717940 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.717904    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08bf8963-4514-481e-99d1-c2cf3eaf88cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bf8963-4514-481e-99d1-c2cf3eaf88cb-plugins-config-volume\") pod \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\" (UID: \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\") "
Nov 02 15:16:45.718129 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.717957    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bf8963-4514-481e-99d1-c2cf3eaf88cb-kube-api-access-9fctc\" (UniqueName: \"kubernetes.io/projected/08bf8963-4514-481e-99d1-c2cf3eaf88cb-kube-api-access-9fctc\") pod \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\" (UID: \"08bf8963-4514-481e-99d1-c2cf3eaf88cb\") "
Nov 02 15:16:45.718341 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.718317    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bf8963-4514-481e-99d1-c2cf3eaf88cb-plugins-config-volume" pod "08bf8963-4514-481e-99d1-c2cf3eaf88cb" (UID: "08bf8963-4514-481e-99d1-c2cf3eaf88cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:16:45.718439 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.718406    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82b074ff-015b-4d8b-92ec-60d3cc4e2778-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:45.718439 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.718426    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb8ll\" (UniqueName: \"kubernetes.io/projected/82b074ff-015b-4d8b-92ec-60d3cc4e2778-kube-api-access-cb8ll\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:45.720667 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.720646    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bf8963-4514-481e-99d1-c2cf3eaf88cb-kube-api-access-9fctc" pod "08bf8963-4514-481e-99d1-c2cf3eaf88cb" (UID: "08bf8963-4514-481e-99d1-c2cf3eaf88cb"). InnerVolumeSpecName "kube-api-access-9fctc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:16:45.819171 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.819095    5034 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bf8963-4514-481e-99d1-c2cf3eaf88cb-plugins-config-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:45.819171 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:45.819124    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fctc\" (UniqueName: \"kubernetes.io/projected/08bf8963-4514-481e-99d1-c2cf3eaf88cb-kube-api-access-9fctc\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:16:46.366315 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.366280    5034 generic.go:356] "Generic (PLEG): container finished" podID="82b074ff-015b-4d8b-92ec-60d3cc4e2778" containerID="5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87" exitCode=0
Nov 02 15:16:46.366724 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.366329    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"
Nov 02 15:16:46.366724 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.366367    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b" event={"ID":"82b074ff-015b-4d8b-92ec-60d3cc4e2778","Type":"ContainerDied","Data":"5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87"}
Nov 02 15:16:46.366724 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.366421    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b" event={"ID":"82b074ff-015b-4d8b-92ec-60d3cc4e2778","Type":"ContainerDied","Data":"5333ad5558111b96b88a6fb4a4467efcb82561921717b5b21b8efd64bb907de2"}
Nov 02 15:16:46.366724 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.366446    5034 scope.go:122] "RemoveContainer" containerID="5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87"
Nov 02 15:16:46.367918 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.367888    5034 generic.go:356] "Generic (PLEG): container finished" podID="08bf8963-4514-481e-99d1-c2cf3eaf88cb" containerID="c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79" exitCode=0
Nov 02 15:16:46.368051 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.367921    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5" event={"ID":"08bf8963-4514-481e-99d1-c2cf3eaf88cb","Type":"ContainerDied","Data":"c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79"}
Nov 02 15:16:46.368051 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.367954    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"
Nov 02 15:16:46.368051 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.367970    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5" event={"ID":"08bf8963-4514-481e-99d1-c2cf3eaf88cb","Type":"ContainerDied","Data":"daa418927c8f4a5ae9cf559b2d66c9722902f904b19f851111db4cdddfdfb4f5"}
Nov 02 15:16:46.377180 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.377161    5034 scope.go:122] "RemoveContainer" containerID="5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87"
Nov 02 15:16:46.377485 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:16:46.377464    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87\": container with ID starting with 5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87 not found: ID does not exist" containerID="5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87"
Nov 02 15:16:46.377581 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.377490    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87"} err="failed to get container status \"5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87\": rpc error: code = NotFound desc = could not find container \"5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87\": container with ID starting with 5400b4343015345189cbf5dd9de118240f728edb030d0f1c40a18c39a5262d87 not found: ID does not exist"
Nov 02 15:16:46.377581 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.377507    5034 scope.go:122] "RemoveContainer" containerID="c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79"
Nov 02 15:16:46.387104 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.387085    5034 scope.go:122] "RemoveContainer" containerID="c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79"
Nov 02 15:16:46.387379 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:16:46.387358    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79\": container with ID starting with c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79 not found: ID does not exist" containerID="c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79"
Nov 02 15:16:46.387481 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.387388    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79"} err="failed to get container status \"c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79\": rpc error: code = NotFound desc = could not find container \"c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79\": container with ID starting with c6cf616cd1d692cf207ea2a39bbea3cf570386daf480b082636bfb7d32540d79 not found: ID does not exist"
Nov 02 15:16:46.421863 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.421842    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"]
Nov 02 15:16:46.434705 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.434684    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-437128-7fd447f952v58b"]
Nov 02 15:16:46.460809 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.460786    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"]
Nov 02 15:16:46.465381 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:46.465360    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-437128-8498dgrrk5"]
Nov 02 15:16:47.392751 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:47.392721    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08bf8963-4514-481e-99d1-c2cf3eaf88cb" path="/var/lib/kubelet/pods/08bf8963-4514-481e-99d1-c2cf3eaf88cb/volumes"
Nov 02 15:16:47.393160 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:16:47.393119    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b074ff-015b-4d8b-92ec-60d3cc4e2778" path="/var/lib/kubelet/pods/82b074ff-015b-4d8b-92ec-60d3cc4e2778/volumes"
Nov 02 15:16:48.418141 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:16:48.418102    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:17:02.795932 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:02.795897    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"]
Nov 02 15:17:02.796398 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:02.796107    5034 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h" podUID="58393844-0472-4a85-ba31-9586f62ea2c2" containerName="manager" containerID="cri-o://4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f" gracePeriod=10
Nov 02 15:17:04.638888 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:04.638864    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 15:17:04.740681 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:04.740648    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58393844-0472-4a85-ba31-9586f62ea2c2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58393844-0472-4a85-ba31-9586f62ea2c2-extensions-socket-volume\") pod \"58393844-0472-4a85-ba31-9586f62ea2c2\" (UID: \"58393844-0472-4a85-ba31-9586f62ea2c2\") "
Nov 02 15:17:04.740890 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:04.740874    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58393844-0472-4a85-ba31-9586f62ea2c2-kube-api-access-k2d6c\" (UniqueName: \"kubernetes.io/projected/58393844-0472-4a85-ba31-9586f62ea2c2-kube-api-access-k2d6c\") pod \"58393844-0472-4a85-ba31-9586f62ea2c2\" (UID: \"58393844-0472-4a85-ba31-9586f62ea2c2\") "
Nov 02 15:17:04.741051 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:04.741028    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58393844-0472-4a85-ba31-9586f62ea2c2-extensions-socket-volume" pod "58393844-0472-4a85-ba31-9586f62ea2c2" (UID: "58393844-0472-4a85-ba31-9586f62ea2c2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 15:17:04.741205 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:04.741189    5034 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58393844-0472-4a85-ba31-9586f62ea2c2-extensions-socket-volume\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:17:04.743120 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:04.743090    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58393844-0472-4a85-ba31-9586f62ea2c2-kube-api-access-k2d6c" pod "58393844-0472-4a85-ba31-9586f62ea2c2" (UID: "58393844-0472-4a85-ba31-9586f62ea2c2"). InnerVolumeSpecName "kube-api-access-k2d6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:17:04.842519 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:04.842491    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2d6c\" (UniqueName: \"kubernetes.io/projected/58393844-0472-4a85-ba31-9586f62ea2c2-kube-api-access-k2d6c\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:17:05.480973 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:05.480939    5034 generic.go:356] "Generic (PLEG): container finished" podID="58393844-0472-4a85-ba31-9586f62ea2c2" containerID="4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f" exitCode=0
Nov 02 15:17:05.481132 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:05.480986    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"
Nov 02 15:17:05.481132 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:05.481022    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h" event={"ID":"58393844-0472-4a85-ba31-9586f62ea2c2","Type":"ContainerDied","Data":"4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f"}
Nov 02 15:17:05.481132 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:05.481074    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h" event={"ID":"58393844-0472-4a85-ba31-9586f62ea2c2","Type":"ContainerDied","Data":"820e6b58586736a5194c327191978a9d685e183c1993d499cf54f907735372c3"}
Nov 02 15:17:05.481132 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:05.481095    5034 scope.go:122] "RemoveContainer" containerID="4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f"
Nov 02 15:17:05.492804 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:05.492769    5034 scope.go:122] "RemoveContainer" containerID="4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f"
Nov 02 15:17:05.493046 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:17:05.493028    5034 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f\": container with ID starting with 4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f not found: ID does not exist" containerID="4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f"
Nov 02 15:17:05.493105 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:05.493054    5034 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f"} err="failed to get container status \"4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f\": rpc error: code = NotFound desc = could not find container \"4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f\": container with ID starting with 4ef87a885fc7917e005afdcd40c708a9970f1aa372e7d428bb6363ff23becd7f not found: ID does not exist"
Nov 02 15:17:05.548130 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:05.548103    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"]
Nov 02 15:17:05.561176 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:05.561154    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bwz2h"]
Nov 02 15:17:07.393014 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:17:07.392983    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58393844-0472-4a85-ba31-9586f62ea2c2" path="/var/lib/kubelet/pods/58393844-0472-4a85-ba31-9586f62ea2c2/volumes"
Nov 02 15:17:18.415126 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:17:18.415024    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:17:48.412521 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:17:48.412476    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:18:05.835302 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835267    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"]
Nov 02 15:18:05.835770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835691    5034 cpu_manager.go:414] "containerMap: removing container" podUID="58393844-0472-4a85-ba31-9586f62ea2c2" containerName="manager"
Nov 02 15:18:05.835770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835704    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58393844-0472-4a85-ba31-9586f62ea2c2" containerName="manager"
Nov 02 15:18:05.835770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835723    5034 cpu_manager.go:414] "containerMap: removing container" podUID="08bf8963-4514-481e-99d1-c2cf3eaf88cb" containerName="payload-pre-processing"
Nov 02 15:18:05.835770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835730    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bf8963-4514-481e-99d1-c2cf3eaf88cb" containerName="payload-pre-processing"
Nov 02 15:18:05.835770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835741    5034 cpu_manager.go:414] "containerMap: removing container" podUID="82b074ff-015b-4d8b-92ec-60d3cc4e2778" containerName="payload-processing"
Nov 02 15:18:05.835770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835749    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b074ff-015b-4d8b-92ec-60d3cc4e2778" containerName="payload-processing"
Nov 02 15:18:05.835770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835773    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f81fb52d-c852-49c8-b8cc-2ccc087ff260" containerName="revoke-keys"
Nov 02 15:18:05.836036 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835793    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81fb52d-c852-49c8-b8cc-2ccc087ff260" containerName="revoke-keys"
Nov 02 15:18:05.836036 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835898    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81fb52d-c852-49c8-b8cc-2ccc087ff260" containerName="revoke-keys"
Nov 02 15:18:05.836036 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835918    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="58393844-0472-4a85-ba31-9586f62ea2c2" containerName="manager"
Nov 02 15:18:05.836036 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835925    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="08bf8963-4514-481e-99d1-c2cf3eaf88cb" containerName="payload-pre-processing"
Nov 02 15:18:05.836036 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.835931    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b074ff-015b-4d8b-92ec-60d3cc4e2778" containerName="payload-processing"
Nov 02 15:18:05.839093 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.839077    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:05.853662 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.853642    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"]
Nov 02 15:18:05.865237 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.865219    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m76wd\""
Nov 02 15:18:05.997997 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.997966    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a960adf1-ea95-4438-a3dd-fff9cbb60dbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7vxsk\" (UID: \"a960adf1-ea95-4438-a3dd-fff9cbb60dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:05.998159 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:05.998116    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795fz\" (UniqueName: \"kubernetes.io/projected/a960adf1-ea95-4438-a3dd-fff9cbb60dbd-kube-api-access-795fz\") pod \"kuadrant-operator-controller-manager-678d87766f-7vxsk\" (UID: \"a960adf1-ea95-4438-a3dd-fff9cbb60dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:06.098890 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.098808    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795fz\" (UniqueName: \"kubernetes.io/projected/a960adf1-ea95-4438-a3dd-fff9cbb60dbd-kube-api-access-795fz\") pod \"kuadrant-operator-controller-manager-678d87766f-7vxsk\" (UID: \"a960adf1-ea95-4438-a3dd-fff9cbb60dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:06.098890 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.098865    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a960adf1-ea95-4438-a3dd-fff9cbb60dbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7vxsk\" (UID: \"a960adf1-ea95-4438-a3dd-fff9cbb60dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:06.099386 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.099366    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a960adf1-ea95-4438-a3dd-fff9cbb60dbd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7vxsk\" (UID: \"a960adf1-ea95-4438-a3dd-fff9cbb60dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:06.123522 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.123498    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-795fz\" (UniqueName: \"kubernetes.io/projected/a960adf1-ea95-4438-a3dd-fff9cbb60dbd-kube-api-access-795fz\") pod \"kuadrant-operator-controller-manager-678d87766f-7vxsk\" (UID: \"a960adf1-ea95-4438-a3dd-fff9cbb60dbd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:06.151189 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.151165    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:06.479602 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.479577    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"]
Nov 02 15:18:06.479748 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:18:06.479730    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda960adf1_ea95_4438_a3dd_fff9cbb60dbd.slice/crio-1f47005f2a58cf1ca1cce3611c5d6f3b9a129c04e9d5b82c1491e434250c841c WatchSource:0}: Error finding container 1f47005f2a58cf1ca1cce3611c5d6f3b9a129c04e9d5b82c1491e434250c841c: Status 404 returned error can't find the container with id 1f47005f2a58cf1ca1cce3611c5d6f3b9a129c04e9d5b82c1491e434250c841c
Nov 02 15:18:06.776540 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.776502    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk" event={"ID":"a960adf1-ea95-4438-a3dd-fff9cbb60dbd","Type":"ContainerStarted","Data":"549067a9d844551df5bf5113e36d46d423c616ca21954b713762a4cd904b8309"}
Nov 02 15:18:06.776709 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.776560    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk" event={"ID":"a960adf1-ea95-4438-a3dd-fff9cbb60dbd","Type":"ContainerStarted","Data":"1f47005f2a58cf1ca1cce3611c5d6f3b9a129c04e9d5b82c1491e434250c841c"}
Nov 02 15:18:06.776754 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.776719    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:06.826712 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:06.826659    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk" podStartSLOduration=1.8266470099999998 podStartE2EDuration="1.82664701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:06.825471608 +0000 UTC m=+5677.576776045" watchObservedRunningTime="2026-11-02 15:18:06.82664701 +0000 UTC m=+5677.577951447"
Nov 02 15:18:17.783512 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:17.783483    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7vxsk"
Nov 02 15:18:18.412384 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:18:18.412355    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:18:31.233871 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:31.233823    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:18:31.236449 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:18:31.236410    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:18:48.594321 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:18:48.594114    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:19:18.412623 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:19:18.412582    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:19:23.846341 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:23.846298    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:19:23.847089 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:23.847052    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:19:28.065793 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.065739    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"]
Nov 02 15:19:28.069497 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.069477    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.086877 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.086849    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:19:28.088995 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.088972    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"]
Nov 02 15:19:28.099789 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.099749    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:19:28.099912 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.099791    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:19:28.099912 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.099806    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:19:28.142254 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.142224    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c5a242a-4mxdk\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.142359 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.142322    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74jsz\" (UniqueName: \"kubernetes.io/projected/a2944a2e-3222-4ca7-9da1-6e33855f65a0-kube-api-access-74jsz\") pod \"maas-api-revoke-sub-2c5a242a-4mxdk\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.142403 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.142386    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c5a242a-4mxdk\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.242819 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.242764    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c5a242a-4mxdk\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.242922 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.242856    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74jsz\" (UniqueName: \"kubernetes.io/projected/a2944a2e-3222-4ca7-9da1-6e33855f65a0-kube-api-access-74jsz\") pod \"maas-api-revoke-sub-2c5a242a-4mxdk\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.242922 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.242907    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c5a242a-4mxdk\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.243458 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.243433    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c5a242a-4mxdk\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.245021 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.244997    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c5a242a-4mxdk\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.267128 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.267102    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74jsz\" (UniqueName: \"kubernetes.io/projected/a2944a2e-3222-4ca7-9da1-6e33855f65a0-kube-api-access-74jsz\") pod \"maas-api-revoke-sub-2c5a242a-4mxdk\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.383960 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.383910    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:28.681130 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:28.681100    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"]
Nov 02 15:19:28.681611 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:19:28.681585    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2944a2e_3222_4ca7_9da1_6e33855f65a0.slice/crio-04f616ce3d75b1e4beea16b4a115cb86bbae1e95125d3f8cb2c1b0791652abb5 WatchSource:0}: Error finding container 04f616ce3d75b1e4beea16b4a115cb86bbae1e95125d3f8cb2c1b0791652abb5: Status 404 returned error can't find the container with id 04f616ce3d75b1e4beea16b4a115cb86bbae1e95125d3f8cb2c1b0791652abb5
Nov 02 15:19:29.320971 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:29.320938    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk" event={"ID":"a2944a2e-3222-4ca7-9da1-6e33855f65a0","Type":"ContainerStarted","Data":"f08223b96156eff039f41ed0d373ea1dcbc6ba4672859d6d6e1a234b72760466"}
Nov 02 15:19:29.321428 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:29.320987    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk" event={"ID":"a2944a2e-3222-4ca7-9da1-6e33855f65a0","Type":"ContainerStarted","Data":"04f616ce3d75b1e4beea16b4a115cb86bbae1e95125d3f8cb2c1b0791652abb5"}
Nov 02 15:19:29.364170 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:29.364123    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk" podStartSLOduration=1.364109179 podStartE2EDuration="1.364109179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:29.363077705 +0000 UTC m=+5760.114382141" watchObservedRunningTime="2026-11-02 15:19:29.364109179 +0000 UTC m=+5760.115413615"
Nov 02 15:19:30.333511 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:30.333470    5034 generic.go:356] "Generic (PLEG): container finished" podID="a2944a2e-3222-4ca7-9da1-6e33855f65a0" containerID="f08223b96156eff039f41ed0d373ea1dcbc6ba4672859d6d6e1a234b72760466" exitCode=0
Nov 02 15:19:30.334041 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:30.333552    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk" event={"ID":"a2944a2e-3222-4ca7-9da1-6e33855f65a0","Type":"ContainerDied","Data":"f08223b96156eff039f41ed0d373ea1dcbc6ba4672859d6d6e1a234b72760466"}
Nov 02 15:19:31.477076 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.477053    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:31.568679 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.568650    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2944a2e-3222-4ca7-9da1-6e33855f65a0-kube-api-access-74jsz\" (UniqueName: \"kubernetes.io/projected/a2944a2e-3222-4ca7-9da1-6e33855f65a0-kube-api-access-74jsz\") pod \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") "
Nov 02 15:19:31.568851 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.568763    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-xks-ca\") pod \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") "
Nov 02 15:19:31.568913 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.568854    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-openshift-ca\") pod \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\" (UID: \"a2944a2e-3222-4ca7-9da1-6e33855f65a0\") "
Nov 02 15:19:31.569201 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.569176    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-openshift-ca" pod "a2944a2e-3222-4ca7-9da1-6e33855f65a0" (UID: "a2944a2e-3222-4ca7-9da1-6e33855f65a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:31.570667 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.570645    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-xks-ca" pod "a2944a2e-3222-4ca7-9da1-6e33855f65a0" (UID: "a2944a2e-3222-4ca7-9da1-6e33855f65a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:19:31.570761 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.570742    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2944a2e-3222-4ca7-9da1-6e33855f65a0-kube-api-access-74jsz" pod "a2944a2e-3222-4ca7-9da1-6e33855f65a0" (UID: "a2944a2e-3222-4ca7-9da1-6e33855f65a0"). InnerVolumeSpecName "kube-api-access-74jsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:31.669531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.669470    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74jsz\" (UniqueName: \"kubernetes.io/projected/a2944a2e-3222-4ca7-9da1-6e33855f65a0-kube-api-access-74jsz\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:19:31.669531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.669502    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:19:31.669531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:31.669511    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2944a2e-3222-4ca7-9da1-6e33855f65a0-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:19:32.344969 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:32.344936    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"
Nov 02 15:19:32.345149 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:32.344973    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk" event={"ID":"a2944a2e-3222-4ca7-9da1-6e33855f65a0","Type":"ContainerDied","Data":"04f616ce3d75b1e4beea16b4a115cb86bbae1e95125d3f8cb2c1b0791652abb5"}
Nov 02 15:19:32.345149 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:32.345025    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04f616ce3d75b1e4beea16b4a115cb86bbae1e95125d3f8cb2c1b0791652abb5"
Nov 02 15:19:38.401745 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.401711    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"]
Nov 02 15:19:38.402235 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.402214    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a2944a2e-3222-4ca7-9da1-6e33855f65a0" containerName="revoke-keys"
Nov 02 15:19:38.402235 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.402234    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2944a2e-3222-4ca7-9da1-6e33855f65a0" containerName="revoke-keys"
Nov 02 15:19:38.402408 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.402332    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2944a2e-3222-4ca7-9da1-6e33855f65a0" containerName="revoke-keys"
Nov 02 15:19:38.405580 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.405563    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.421850 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.421823    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929198de-86277\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.422000 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.421869    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929198de-86277\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.422000 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.421920    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlzzs\" (UniqueName: \"kubernetes.io/projected/39a85145-3ee8-48b4-95ae-601acdf62a19-kube-api-access-nlzzs\") pod \"maas-api-revoke-sub-929198de-86277\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.426000 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.425975    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"]
Nov 02 15:19:38.435748 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.435721    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:19:38.435890 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.435770    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:19:38.435890 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.435865    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:19:38.436021 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.435890    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:19:38.522958 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.522928    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929198de-86277\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.523126 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.522965    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929198de-86277\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.523126 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.522998    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlzzs\" (UniqueName: \"kubernetes.io/projected/39a85145-3ee8-48b4-95ae-601acdf62a19-kube-api-access-nlzzs\") pod \"maas-api-revoke-sub-929198de-86277\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.523571 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.523552    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-929198de-86277\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.525341 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.525313    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-929198de-86277\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.547336 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.547313    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlzzs\" (UniqueName: \"kubernetes.io/projected/39a85145-3ee8-48b4-95ae-601acdf62a19-kube-api-access-nlzzs\") pod \"maas-api-revoke-sub-929198de-86277\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:38.718727 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:38.718663    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:39.018838 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:19:39.018808    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a85145_3ee8_48b4_95ae_601acdf62a19.slice/crio-7689ce118dbda1eaa24f03f76d712d24d2fa5c74569527e02803a74668fb3597 WatchSource:0}: Error finding container 7689ce118dbda1eaa24f03f76d712d24d2fa5c74569527e02803a74668fb3597: Status 404 returned error can't find the container with id 7689ce118dbda1eaa24f03f76d712d24d2fa5c74569527e02803a74668fb3597
Nov 02 15:19:39.018988 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:39.018897    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"]
Nov 02 15:19:39.394887 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:39.394763    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277" event={"ID":"39a85145-3ee8-48b4-95ae-601acdf62a19","Type":"ContainerStarted","Data":"c10b43294686bd0b34fb450ee406cee1f0948177019d83522774496e3064cff7"}
Nov 02 15:19:39.394887 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:39.394839    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277" event={"ID":"39a85145-3ee8-48b4-95ae-601acdf62a19","Type":"ContainerStarted","Data":"7689ce118dbda1eaa24f03f76d712d24d2fa5c74569527e02803a74668fb3597"}
Nov 02 15:19:39.437671 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:39.437621    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277" podStartSLOduration=1.437608966 podStartE2EDuration="1.437608966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:39.43590166 +0000 UTC m=+5770.187206098" watchObservedRunningTime="2026-11-02 15:19:39.437608966 +0000 UTC m=+5770.188913402"
Nov 02 15:19:40.403061 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:40.403018    5034 generic.go:356] "Generic (PLEG): container finished" podID="39a85145-3ee8-48b4-95ae-601acdf62a19" containerID="c10b43294686bd0b34fb450ee406cee1f0948177019d83522774496e3064cff7" exitCode=0
Nov 02 15:19:40.403254 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:40.403101    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277" event={"ID":"39a85145-3ee8-48b4-95ae-601acdf62a19","Type":"ContainerDied","Data":"c10b43294686bd0b34fb450ee406cee1f0948177019d83522774496e3064cff7"}
Nov 02 15:19:41.539589 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.539566    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:41.552706 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.552682    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-openshift-ca\") pod \"39a85145-3ee8-48b4-95ae-601acdf62a19\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") "
Nov 02 15:19:41.552867 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.552773    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a85145-3ee8-48b4-95ae-601acdf62a19-kube-api-access-nlzzs\" (UniqueName: \"kubernetes.io/projected/39a85145-3ee8-48b4-95ae-601acdf62a19-kube-api-access-nlzzs\") pod \"39a85145-3ee8-48b4-95ae-601acdf62a19\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") "
Nov 02 15:19:41.552867 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.552833    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-xks-ca\") pod \"39a85145-3ee8-48b4-95ae-601acdf62a19\" (UID: \"39a85145-3ee8-48b4-95ae-601acdf62a19\") "
Nov 02 15:19:41.553088 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.553064    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-openshift-ca" pod "39a85145-3ee8-48b4-95ae-601acdf62a19" (UID: "39a85145-3ee8-48b4-95ae-601acdf62a19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:19:41.553195 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.553182    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:19:41.554735 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.554702    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a85145-3ee8-48b4-95ae-601acdf62a19-kube-api-access-nlzzs" pod "39a85145-3ee8-48b4-95ae-601acdf62a19" (UID: "39a85145-3ee8-48b4-95ae-601acdf62a19"). InnerVolumeSpecName "kube-api-access-nlzzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:19:41.554735 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.554719    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-xks-ca" pod "39a85145-3ee8-48b4-95ae-601acdf62a19" (UID: "39a85145-3ee8-48b4-95ae-601acdf62a19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:19:41.654445 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.654419    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlzzs\" (UniqueName: \"kubernetes.io/projected/39a85145-3ee8-48b4-95ae-601acdf62a19-kube-api-access-nlzzs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:19:41.654445 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:41.654446    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39a85145-3ee8-48b4-95ae-601acdf62a19-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:19:42.414207 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:42.414180    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"
Nov 02 15:19:42.414382 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:42.414176    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277" event={"ID":"39a85145-3ee8-48b4-95ae-601acdf62a19","Type":"ContainerDied","Data":"7689ce118dbda1eaa24f03f76d712d24d2fa5c74569527e02803a74668fb3597"}
Nov 02 15:19:42.414382 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:19:42.414301    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7689ce118dbda1eaa24f03f76d712d24d2fa5c74569527e02803a74668fb3597"
Nov 02 15:19:48.411860 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:19:48.411823    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:20:00.075131 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.075095    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"]
Nov 02 15:20:00.078030 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.075525    5034 cpu_manager.go:414] "containerMap: removing container" podUID="39a85145-3ee8-48b4-95ae-601acdf62a19" containerName="revoke-keys"
Nov 02 15:20:00.078030 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.075535    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a85145-3ee8-48b4-95ae-601acdf62a19" containerName="revoke-keys"
Nov 02 15:20:00.078030 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.075618    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a85145-3ee8-48b4-95ae-601acdf62a19" containerName="revoke-keys"
Nov 02 15:20:00.078897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.078873    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.099142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.098764    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"]
Nov 02 15:20:00.111815 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.109185    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:20:00.111815 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.109211    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:20:00.111815 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.109234    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:20:00.111815 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.109286    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:20:00.159008 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.158981    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0cfa585-bwlmn\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.159155 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.159109    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0cfa585-bwlmn\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.159200 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.159164    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c82q9\" (UniqueName: \"kubernetes.io/projected/90f5db57-6b4e-445e-be97-a468f73b9511-kube-api-access-c82q9\") pod \"maas-api-revoke-sub-e0cfa585-bwlmn\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.260313 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.260282    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0cfa585-bwlmn\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.260441 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.260327    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c82q9\" (UniqueName: \"kubernetes.io/projected/90f5db57-6b4e-445e-be97-a468f73b9511-kube-api-access-c82q9\") pod \"maas-api-revoke-sub-e0cfa585-bwlmn\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.260441 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.260360    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0cfa585-bwlmn\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.261103 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.261082    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0cfa585-bwlmn\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.262567 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.262548    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0cfa585-bwlmn\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.284618 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.284593    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c82q9\" (UniqueName: \"kubernetes.io/projected/90f5db57-6b4e-445e-be97-a468f73b9511-kube-api-access-c82q9\") pod \"maas-api-revoke-sub-e0cfa585-bwlmn\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.395458 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.395394    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:00.692335 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:20:00.692308    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f5db57_6b4e_445e_be97_a468f73b9511.slice/crio-e6d36df42718d11542d0c5cb441d75d7ff6ebf4596a3f2f42707d50e8f2e3630 WatchSource:0}: Error finding container e6d36df42718d11542d0c5cb441d75d7ff6ebf4596a3f2f42707d50e8f2e3630: Status 404 returned error can't find the container with id e6d36df42718d11542d0c5cb441d75d7ff6ebf4596a3f2f42707d50e8f2e3630
Nov 02 15:20:00.692489 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:00.692466    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"]
Nov 02 15:20:01.528163 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:01.528131    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn" event={"ID":"90f5db57-6b4e-445e-be97-a468f73b9511","Type":"ContainerStarted","Data":"ab3fb765124edb7fde55598dcd453650808324e29041064ccfbe64931bbf4355"}
Nov 02 15:20:01.528534 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:01.528183    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn" event={"ID":"90f5db57-6b4e-445e-be97-a468f73b9511","Type":"ContainerStarted","Data":"e6d36df42718d11542d0c5cb441d75d7ff6ebf4596a3f2f42707d50e8f2e3630"}
Nov 02 15:20:01.571974 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:01.571921    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn" podStartSLOduration=1.5719089450000001 podStartE2EDuration="1.571908945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:01.569495242 +0000 UTC m=+5792.320799699" watchObservedRunningTime="2026-11-02 15:20:01.571908945 +0000 UTC m=+5792.323213414"
Nov 02 15:20:02.534184 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:02.534148    5034 generic.go:356] "Generic (PLEG): container finished" podID="90f5db57-6b4e-445e-be97-a468f73b9511" containerID="ab3fb765124edb7fde55598dcd453650808324e29041064ccfbe64931bbf4355" exitCode=0
Nov 02 15:20:02.534643 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:02.534220    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn" event={"ID":"90f5db57-6b4e-445e-be97-a468f73b9511","Type":"ContainerDied","Data":"ab3fb765124edb7fde55598dcd453650808324e29041064ccfbe64931bbf4355"}
Nov 02 15:20:03.671927 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.671901    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:03.792389 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.792315    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f5db57-6b4e-445e-be97-a468f73b9511-kube-api-access-c82q9\" (UniqueName: \"kubernetes.io/projected/90f5db57-6b4e-445e-be97-a468f73b9511-kube-api-access-c82q9\") pod \"90f5db57-6b4e-445e-be97-a468f73b9511\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") "
Nov 02 15:20:03.792389 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.792358    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-xks-ca\") pod \"90f5db57-6b4e-445e-be97-a468f73b9511\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") "
Nov 02 15:20:03.792576 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.792437    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-openshift-ca\") pod \"90f5db57-6b4e-445e-be97-a468f73b9511\" (UID: \"90f5db57-6b4e-445e-be97-a468f73b9511\") "
Nov 02 15:20:03.792841 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.792812    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-openshift-ca" pod "90f5db57-6b4e-445e-be97-a468f73b9511" (UID: "90f5db57-6b4e-445e-be97-a468f73b9511"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:03.794364 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.794334    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f5db57-6b4e-445e-be97-a468f73b9511-kube-api-access-c82q9" pod "90f5db57-6b4e-445e-be97-a468f73b9511" (UID: "90f5db57-6b4e-445e-be97-a468f73b9511"). InnerVolumeSpecName "kube-api-access-c82q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:03.794479 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.794429    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-xks-ca" pod "90f5db57-6b4e-445e-be97-a468f73b9511" (UID: "90f5db57-6b4e-445e-be97-a468f73b9511"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:03.893868 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.893841    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c82q9\" (UniqueName: \"kubernetes.io/projected/90f5db57-6b4e-445e-be97-a468f73b9511-kube-api-access-c82q9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:20:03.893868 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.893863    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:20:03.894030 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:03.893872    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f5db57-6b4e-445e-be97-a468f73b9511-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:20:04.545021 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:04.544990    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"
Nov 02 15:20:04.545201 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:04.545022    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn" event={"ID":"90f5db57-6b4e-445e-be97-a468f73b9511","Type":"ContainerDied","Data":"e6d36df42718d11542d0c5cb441d75d7ff6ebf4596a3f2f42707d50e8f2e3630"}
Nov 02 15:20:04.545201 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:04.545070    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6d36df42718d11542d0c5cb441d75d7ff6ebf4596a3f2f42707d50e8f2e3630"
Nov 02 15:20:18.534629 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:20:18.534426    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:20:30.686233 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.686199    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"]
Nov 02 15:20:30.686641 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.686621    5034 cpu_manager.go:414] "containerMap: removing container" podUID="90f5db57-6b4e-445e-be97-a468f73b9511" containerName="revoke-keys"
Nov 02 15:20:30.686641 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.686632    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f5db57-6b4e-445e-be97-a468f73b9511" containerName="revoke-keys"
Nov 02 15:20:30.686737 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.686727    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f5db57-6b4e-445e-be97-a468f73b9511" containerName="revoke-keys"
Nov 02 15:20:30.689965 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.689944    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:30.720276 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.720250    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:20:30.720495 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.720475    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:20:30.720633 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.720506    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:20:30.720633 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.720522    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:20:30.722747 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.722722    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"]
Nov 02 15:20:30.778029 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.778001    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c06e0c99-qp9nd\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:30.778029 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.778032    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x85n\" (UniqueName: \"kubernetes.io/projected/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-kube-api-access-5x85n\") pod \"maas-api-revoke-sub-c06e0c99-qp9nd\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:30.778231 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.778162    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c06e0c99-qp9nd\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:30.878949 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.878913    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c06e0c99-qp9nd\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:30.878949 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.878955    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x85n\" (UniqueName: \"kubernetes.io/projected/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-kube-api-access-5x85n\") pod \"maas-api-revoke-sub-c06e0c99-qp9nd\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:30.879332 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.879038    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c06e0c99-qp9nd\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:30.879747 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.879720    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c06e0c99-qp9nd\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:30.881386 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.881363    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c06e0c99-qp9nd\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:30.903192 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:30.903166    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x85n\" (UniqueName: \"kubernetes.io/projected/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-kube-api-access-5x85n\") pod \"maas-api-revoke-sub-c06e0c99-qp9nd\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:31.005995 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:31.005962    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:31.307605 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:31.307575    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"]
Nov 02 15:20:31.308090 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:20:31.308063    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b95aad_3cb3_4d1c_ba81_948eac5db77d.slice/crio-d80fb789697c88a708249b2e7feca3205ece7b7d12d2708b1ec9ef6131cd1603 WatchSource:0}: Error finding container d80fb789697c88a708249b2e7feca3205ece7b7d12d2708b1ec9ef6131cd1603: Status 404 returned error can't find the container with id d80fb789697c88a708249b2e7feca3205ece7b7d12d2708b1ec9ef6131cd1603
Nov 02 15:20:31.759495 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:31.759459    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd" event={"ID":"c8b95aad-3cb3-4d1c-ba81-948eac5db77d","Type":"ContainerStarted","Data":"ec9ed564a9c9cc45c8b0a236496164f3e73c8881e2badd09dc1fb4b53260d85e"}
Nov 02 15:20:31.759926 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:31.759540    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd" event={"ID":"c8b95aad-3cb3-4d1c-ba81-948eac5db77d","Type":"ContainerStarted","Data":"d80fb789697c88a708249b2e7feca3205ece7b7d12d2708b1ec9ef6131cd1603"}
Nov 02 15:20:31.803463 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:31.803415    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd" podStartSLOduration=1.803402892 podStartE2EDuration="1.803402892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:31.801659943 +0000 UTC m=+5822.552964395" watchObservedRunningTime="2026-11-02 15:20:31.803402892 +0000 UTC m=+5822.554707328"
Nov 02 15:20:32.029326 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:32.029292    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"]
Nov 02 15:20:32.032360 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:32.032337    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e2ebb5d-5z2d6"]
Nov 02 15:20:32.765871 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:32.765831    5034 generic.go:356] "Generic (PLEG): container finished" podID="c8b95aad-3cb3-4d1c-ba81-948eac5db77d" containerID="ec9ed564a9c9cc45c8b0a236496164f3e73c8881e2badd09dc1fb4b53260d85e" exitCode=0
Nov 02 15:20:32.766291 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:32.765912    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd" event={"ID":"c8b95aad-3cb3-4d1c-ba81-948eac5db77d","Type":"ContainerDied","Data":"ec9ed564a9c9cc45c8b0a236496164f3e73c8881e2badd09dc1fb4b53260d85e"}
Nov 02 15:20:33.393201 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:33.393168    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26a35f36-8bab-4fd4-aa74-4cb2073b92c7" path="/var/lib/kubelet/pods/26a35f36-8bab-4fd4-aa74-4cb2073b92c7/volumes"
Nov 02 15:20:33.924751 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:33.924728    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:34.011742 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.011709    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-openshift-ca\") pod \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") "
Nov 02 15:20:34.011947 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.011761    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-xks-ca\") pod \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") "
Nov 02 15:20:34.011947 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.011885    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-kube-api-access-5x85n\" (UniqueName: \"kubernetes.io/projected/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-kube-api-access-5x85n\") pod \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\" (UID: \"c8b95aad-3cb3-4d1c-ba81-948eac5db77d\") "
Nov 02 15:20:34.012147 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.012124    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-openshift-ca" pod "c8b95aad-3cb3-4d1c-ba81-948eac5db77d" (UID: "c8b95aad-3cb3-4d1c-ba81-948eac5db77d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:34.012251 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.012228    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:20:34.013890 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.013863    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-xks-ca" pod "c8b95aad-3cb3-4d1c-ba81-948eac5db77d" (UID: "c8b95aad-3cb3-4d1c-ba81-948eac5db77d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:34.014007 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.013931    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-kube-api-access-5x85n" pod "c8b95aad-3cb3-4d1c-ba81-948eac5db77d" (UID: "c8b95aad-3cb3-4d1c-ba81-948eac5db77d"). InnerVolumeSpecName "kube-api-access-5x85n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:34.113153 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.113076    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x85n\" (UniqueName: \"kubernetes.io/projected/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-kube-api-access-5x85n\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:20:34.113153 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.113111    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b95aad-3cb3-4d1c-ba81-948eac5db77d-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:20:34.778072 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.778032    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd" event={"ID":"c8b95aad-3cb3-4d1c-ba81-948eac5db77d","Type":"ContainerDied","Data":"d80fb789697c88a708249b2e7feca3205ece7b7d12d2708b1ec9ef6131cd1603"}
Nov 02 15:20:34.778072 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.778061    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"
Nov 02 15:20:34.778072 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:34.778081    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d80fb789697c88a708249b2e7feca3205ece7b7d12d2708b1ec9ef6131cd1603"
Nov 02 15:20:41.026547 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.026509    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"]
Nov 02 15:20:41.027023 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.027008    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c8b95aad-3cb3-4d1c-ba81-948eac5db77d" containerName="revoke-keys"
Nov 02 15:20:41.027079 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.027024    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b95aad-3cb3-4d1c-ba81-948eac5db77d" containerName="revoke-keys"
Nov 02 15:20:41.027121 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.027102    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b95aad-3cb3-4d1c-ba81-948eac5db77d" containerName="revoke-keys"
Nov 02 15:20:41.030704 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.030683    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.048149 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.048131    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:20:41.048279 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.048162    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:20:41.061616 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.061577    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:20:41.062317 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.061580    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:20:41.065289 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.065266    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"]
Nov 02 15:20:41.078557 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.078533    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a387dc28-vtpq4\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.078675 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.078580    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6tjs\" (UniqueName: \"kubernetes.io/projected/1c336041-4989-46c4-b03f-4f3a28e6b6de-kube-api-access-z6tjs\") pod \"maas-api-revoke-sub-a387dc28-vtpq4\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.078675 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.078649    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a387dc28-vtpq4\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.179315 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.179278    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a387dc28-vtpq4\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.179492 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.179346    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6tjs\" (UniqueName: \"kubernetes.io/projected/1c336041-4989-46c4-b03f-4f3a28e6b6de-kube-api-access-z6tjs\") pod \"maas-api-revoke-sub-a387dc28-vtpq4\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.179492 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.179378    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a387dc28-vtpq4\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.180081 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.180055    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a387dc28-vtpq4\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.181660 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.181637    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a387dc28-vtpq4\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.204951 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.204922    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6tjs\" (UniqueName: \"kubernetes.io/projected/1c336041-4989-46c4-b03f-4f3a28e6b6de-kube-api-access-z6tjs\") pod \"maas-api-revoke-sub-a387dc28-vtpq4\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.343605 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.343541    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:41.661321 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:20:41.661295    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c336041_4989_46c4_b03f_4f3a28e6b6de.slice/crio-5f0c023b29e0bd0520f8e920cd1e9e3e65a77b3f1f87e115cd810a972898b5fc WatchSource:0}: Error finding container 5f0c023b29e0bd0520f8e920cd1e9e3e65a77b3f1f87e115cd810a972898b5fc: Status 404 returned error can't find the container with id 5f0c023b29e0bd0520f8e920cd1e9e3e65a77b3f1f87e115cd810a972898b5fc
Nov 02 15:20:41.661953 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.661926    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"]
Nov 02 15:20:41.814724 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:41.814691    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4" event={"ID":"1c336041-4989-46c4-b03f-4f3a28e6b6de","Type":"ContainerStarted","Data":"5f0c023b29e0bd0520f8e920cd1e9e3e65a77b3f1f87e115cd810a972898b5fc"}
Nov 02 15:20:42.821621 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:42.821527    5034 generic.go:356] "Generic (PLEG): container finished" podID="1c336041-4989-46c4-b03f-4f3a28e6b6de" containerID="b1dc210befa1f21ecd324da4191083f771a4f86deab6c774aee49f180c4d0432" exitCode=0
Nov 02 15:20:42.822079 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:42.821612    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4" event={"ID":"1c336041-4989-46c4-b03f-4f3a28e6b6de","Type":"ContainerDied","Data":"b1dc210befa1f21ecd324da4191083f771a4f86deab6c774aee49f180c4d0432"}
Nov 02 15:20:43.980231 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:43.980207    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:44.009971 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.009943    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-xks-ca\") pod \"1c336041-4989-46c4-b03f-4f3a28e6b6de\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") "
Nov 02 15:20:44.010110 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.010011    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-openshift-ca\") pod \"1c336041-4989-46c4-b03f-4f3a28e6b6de\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") "
Nov 02 15:20:44.010110 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.010042    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c336041-4989-46c4-b03f-4f3a28e6b6de-kube-api-access-z6tjs\" (UniqueName: \"kubernetes.io/projected/1c336041-4989-46c4-b03f-4f3a28e6b6de-kube-api-access-z6tjs\") pod \"1c336041-4989-46c4-b03f-4f3a28e6b6de\" (UID: \"1c336041-4989-46c4-b03f-4f3a28e6b6de\") "
Nov 02 15:20:44.010425 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.010390    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-openshift-ca" pod "1c336041-4989-46c4-b03f-4f3a28e6b6de" (UID: "1c336041-4989-46c4-b03f-4f3a28e6b6de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:20:44.012089 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.012059    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c336041-4989-46c4-b03f-4f3a28e6b6de-kube-api-access-z6tjs" pod "1c336041-4989-46c4-b03f-4f3a28e6b6de" (UID: "1c336041-4989-46c4-b03f-4f3a28e6b6de"). InnerVolumeSpecName "kube-api-access-z6tjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:20:44.012089 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.012071    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-xks-ca" pod "1c336041-4989-46c4-b03f-4f3a28e6b6de" (UID: "1c336041-4989-46c4-b03f-4f3a28e6b6de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:20:44.110770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.110702    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6tjs\" (UniqueName: \"kubernetes.io/projected/1c336041-4989-46c4-b03f-4f3a28e6b6de-kube-api-access-z6tjs\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:20:44.110770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.110726    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:20:44.110770 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.110737    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336041-4989-46c4-b03f-4f3a28e6b6de-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:20:44.835497 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.835466    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"
Nov 02 15:20:44.835677 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.835501    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4" event={"ID":"1c336041-4989-46c4-b03f-4f3a28e6b6de","Type":"ContainerDied","Data":"5f0c023b29e0bd0520f8e920cd1e9e3e65a77b3f1f87e115cd810a972898b5fc"}
Nov 02 15:20:44.835677 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:20:44.835559    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f0c023b29e0bd0520f8e920cd1e9e3e65a77b3f1f87e115cd810a972898b5fc"
Nov 02 15:20:48.412219 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:20:48.412187    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:21:08.843798 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:08.842956    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:21:08.843798 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:08.843129    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:08.884022 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:08.883910    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:21:08.884022 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:08.883966    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:08.918136 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:08.917862    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:21:08.918447 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:08.918398    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:08.930815 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:08.929857    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-thsm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:21:08.930815 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:08.929901    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-thsm4" podUID="565989d2-f1a9-4413-a50d-fd54da149761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:12.669511 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.669479    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"]
Nov 02 15:21:12.669980 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.669966    5034 cpu_manager.go:414] "containerMap: removing container" podUID="1c336041-4989-46c4-b03f-4f3a28e6b6de" containerName="revoke-keys"
Nov 02 15:21:12.670029 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.669981    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c336041-4989-46c4-b03f-4f3a28e6b6de" containerName="revoke-keys"
Nov 02 15:21:12.670066 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.670056    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c336041-4989-46c4-b03f-4f3a28e6b6de" containerName="revoke-keys"
Nov 02 15:21:12.673190 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.673173    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.695082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.695055    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:21:12.695254 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.695237    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:21:12.695322 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.695251    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:21:12.708398 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.708360    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:21:12.711318 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.711293    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"]
Nov 02 15:21:12.793056 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.793027    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ca3420-rlvpf\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.793228 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.793069    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ca3420-rlvpf\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.793303 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.793279    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89w5g\" (UniqueName: \"kubernetes.io/projected/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-kube-api-access-89w5g\") pod \"maas-api-revoke-sub-08ca3420-rlvpf\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.894227 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.894187    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89w5g\" (UniqueName: \"kubernetes.io/projected/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-kube-api-access-89w5g\") pod \"maas-api-revoke-sub-08ca3420-rlvpf\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.894382 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.894245    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ca3420-rlvpf\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.894382 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.894274    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ca3420-rlvpf\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.895028 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.895003    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08ca3420-rlvpf\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.896589 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.896573    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08ca3420-rlvpf\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.918382 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.918353    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89w5g\" (UniqueName: \"kubernetes.io/projected/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-kube-api-access-89w5g\") pod \"maas-api-revoke-sub-08ca3420-rlvpf\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:12.986228 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:12.986209    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:13.282835 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:21:13.282806    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121cedb8_2e03_4dd6_87ad_3ee40ba1de6e.slice/crio-51cef82c39e575edff33d92326fe7f3ea5fb09ec323275d596763a4dfda70c81 WatchSource:0}: Error finding container 51cef82c39e575edff33d92326fe7f3ea5fb09ec323275d596763a4dfda70c81: Status 404 returned error can't find the container with id 51cef82c39e575edff33d92326fe7f3ea5fb09ec323275d596763a4dfda70c81
Nov 02 15:21:13.283180 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:13.283156    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"]
Nov 02 15:21:14.064494 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:14.064460    5034 generic.go:356] "Generic (PLEG): container finished" podID="121cedb8-2e03-4dd6-87ad-3ee40ba1de6e" containerID="7ead822d8efa143801aca7d8080790edf3dafaef11fdb895e0e3fc3dd988d076" exitCode=0
Nov 02 15:21:14.064827 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:14.064520    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf" event={"ID":"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e","Type":"ContainerDied","Data":"7ead822d8efa143801aca7d8080790edf3dafaef11fdb895e0e3fc3dd988d076"}
Nov 02 15:21:14.064827 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:14.064568    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf" event={"ID":"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e","Type":"ContainerStarted","Data":"51cef82c39e575edff33d92326fe7f3ea5fb09ec323275d596763a4dfda70c81"}
Nov 02 15:21:15.205997 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.205974    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:15.316788 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.316757    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-openshift-ca\") pod \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") "
Nov 02 15:21:15.316939 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.316812    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-xks-ca\") pod \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") "
Nov 02 15:21:15.316980 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.316960    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-kube-api-access-89w5g\" (UniqueName: \"kubernetes.io/projected/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-kube-api-access-89w5g\") pod \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\" (UID: \"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e\") "
Nov 02 15:21:15.317125 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.317106    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-openshift-ca" pod "121cedb8-2e03-4dd6-87ad-3ee40ba1de6e" (UID: "121cedb8-2e03-4dd6-87ad-3ee40ba1de6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:21:15.317391 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.317372    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:21:15.318818 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.318792    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-xks-ca" pod "121cedb8-2e03-4dd6-87ad-3ee40ba1de6e" (UID: "121cedb8-2e03-4dd6-87ad-3ee40ba1de6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:21:15.318939 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.318921    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-kube-api-access-89w5g" pod "121cedb8-2e03-4dd6-87ad-3ee40ba1de6e" (UID: "121cedb8-2e03-4dd6-87ad-3ee40ba1de6e"). InnerVolumeSpecName "kube-api-access-89w5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:21:15.418125 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.418101    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:21:15.418125 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:15.418126    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89w5g\" (UniqueName: \"kubernetes.io/projected/121cedb8-2e03-4dd6-87ad-3ee40ba1de6e-kube-api-access-89w5g\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:21:16.076591 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:16.076560    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf"
Nov 02 15:21:16.076798 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:16.076602    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08ca3420-rlvpf" event={"ID":"121cedb8-2e03-4dd6-87ad-3ee40ba1de6e","Type":"ContainerDied","Data":"51cef82c39e575edff33d92326fe7f3ea5fb09ec323275d596763a4dfda70c81"}
Nov 02 15:21:16.076798 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:16.076655    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51cef82c39e575edff33d92326fe7f3ea5fb09ec323275d596763a4dfda70c81"
Nov 02 15:21:18.413962 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:21:18.413927    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:21:23.000992 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.000960    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"]
Nov 02 15:21:23.001382 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.001369    5034 cpu_manager.go:414] "containerMap: removing container" podUID="121cedb8-2e03-4dd6-87ad-3ee40ba1de6e" containerName="revoke-keys"
Nov 02 15:21:23.001436 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.001383    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121cedb8-2e03-4dd6-87ad-3ee40ba1de6e" containerName="revoke-keys"
Nov 02 15:21:23.001496 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.001486    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="121cedb8-2e03-4dd6-87ad-3ee40ba1de6e" containerName="revoke-keys"
Nov 02 15:21:23.004717 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.004695    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.024611 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.024583    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"]
Nov 02 15:21:23.035071 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.035042    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:21:23.035071 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.035062    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:21:23.035303 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.035108    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:21:23.035422 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.035396    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:21:23.093351 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.093323    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnrl\" (UniqueName: \"kubernetes.io/projected/cedbad61-38ea-4234-b56c-f18a7edab309-kube-api-access-6vnrl\") pod \"maas-api-revoke-sub-8009dbe9-qbzxf\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.093510 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.093368    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8009dbe9-qbzxf\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.093510 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.093456    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8009dbe9-qbzxf\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.193989 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.193948    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8009dbe9-qbzxf\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.194159 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.194057    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnrl\" (UniqueName: \"kubernetes.io/projected/cedbad61-38ea-4234-b56c-f18a7edab309-kube-api-access-6vnrl\") pod \"maas-api-revoke-sub-8009dbe9-qbzxf\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.194159 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.194082    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8009dbe9-qbzxf\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.194608 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.194584    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8009dbe9-qbzxf\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.196354 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.196332    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8009dbe9-qbzxf\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.219181 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.219151    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnrl\" (UniqueName: \"kubernetes.io/projected/cedbad61-38ea-4234-b56c-f18a7edab309-kube-api-access-6vnrl\") pod \"maas-api-revoke-sub-8009dbe9-qbzxf\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.318696 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.318622    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:23.630619 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:23.630587    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"]
Nov 02 15:21:23.631124 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:21:23.631091    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedbad61_38ea_4234_b56c_f18a7edab309.slice/crio-6ca0d26eed093127407379af75267937064c118bb228af6af45b1e63483cb128 WatchSource:0}: Error finding container 6ca0d26eed093127407379af75267937064c118bb228af6af45b1e63483cb128: Status 404 returned error can't find the container with id 6ca0d26eed093127407379af75267937064c118bb228af6af45b1e63483cb128
Nov 02 15:21:24.121153 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:24.121116    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf" event={"ID":"cedbad61-38ea-4234-b56c-f18a7edab309","Type":"ContainerStarted","Data":"28955e2b979d71532cfc0453a2f79a078f7129d709c9eb42bea6d4d9e7ce6b5c"}
Nov 02 15:21:24.121561 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:24.121171    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf" event={"ID":"cedbad61-38ea-4234-b56c-f18a7edab309","Type":"ContainerStarted","Data":"6ca0d26eed093127407379af75267937064c118bb228af6af45b1e63483cb128"}
Nov 02 15:21:24.169904 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:24.169802    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf" podStartSLOduration=2.169789545 podStartE2EDuration="2.169789545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:24.168080658 +0000 UTC m=+5874.919385095" watchObservedRunningTime="2026-11-02 15:21:24.169789545 +0000 UTC m=+5874.921093978"
Nov 02 15:21:25.132100 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:25.132061    5034 generic.go:356] "Generic (PLEG): container finished" podID="cedbad61-38ea-4234-b56c-f18a7edab309" containerID="28955e2b979d71532cfc0453a2f79a078f7129d709c9eb42bea6d4d9e7ce6b5c" exitCode=0
Nov 02 15:21:25.132702 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:25.132445    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf" event={"ID":"cedbad61-38ea-4234-b56c-f18a7edab309","Type":"ContainerDied","Data":"28955e2b979d71532cfc0453a2f79a078f7129d709c9eb42bea6d4d9e7ce6b5c"}
Nov 02 15:21:26.949526 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:26.949491    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:27.064806 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.064161    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-xks-ca\") pod \"cedbad61-38ea-4234-b56c-f18a7edab309\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") "
Nov 02 15:21:27.064806 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.064236    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cedbad61-38ea-4234-b56c-f18a7edab309-kube-api-access-6vnrl\" (UniqueName: \"kubernetes.io/projected/cedbad61-38ea-4234-b56c-f18a7edab309-kube-api-access-6vnrl\") pod \"cedbad61-38ea-4234-b56c-f18a7edab309\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") "
Nov 02 15:21:27.064806 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.064293    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-openshift-ca\") pod \"cedbad61-38ea-4234-b56c-f18a7edab309\" (UID: \"cedbad61-38ea-4234-b56c-f18a7edab309\") "
Nov 02 15:21:27.070005 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.069975    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-openshift-ca" pod "cedbad61-38ea-4234-b56c-f18a7edab309" (UID: "cedbad61-38ea-4234-b56c-f18a7edab309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:21:27.247803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.247444    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:21:27.271872 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.260274    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-xks-ca" pod "cedbad61-38ea-4234-b56c-f18a7edab309" (UID: "cedbad61-38ea-4234-b56c-f18a7edab309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:21:27.276196 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.275972    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cedbad61-38ea-4234-b56c-f18a7edab309-kube-api-access-6vnrl" pod "cedbad61-38ea-4234-b56c-f18a7edab309" (UID: "cedbad61-38ea-4234-b56c-f18a7edab309"). InnerVolumeSpecName "kube-api-access-6vnrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:21:27.277874 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.276936    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf" event={"ID":"cedbad61-38ea-4234-b56c-f18a7edab309","Type":"ContainerDied","Data":"6ca0d26eed093127407379af75267937064c118bb228af6af45b1e63483cb128"}
Nov 02 15:21:27.277874 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.276999    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ca0d26eed093127407379af75267937064c118bb228af6af45b1e63483cb128"
Nov 02 15:21:27.277874 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.277071    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8009dbe9-qbzxf"
Nov 02 15:21:27.351712 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.351680    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cedbad61-38ea-4234-b56c-f18a7edab309-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:21:27.351984 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:27.351963    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vnrl\" (UniqueName: \"kubernetes.io/projected/cedbad61-38ea-4234-b56c-f18a7edab309-kube-api-access-6vnrl\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:21:38.848913 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:38.848870    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-8glk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:21:38.849579 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:38.849546    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-8glk8" podUID="dcf337cc-18f5-4451-88fe-351cf905dca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:38.894800 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:38.885211    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-kbj7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:21:38.894800 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:38.885280    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-kbj7z" podUID="43892ef8-c4cc-414e-bc41-d1626001a0fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:38.918412 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:38.917880    5034 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-4s5pw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 15:21:38.918645 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:38.918617    5034 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-4s5pw" podUID="6b68886e-53d7-470d-8d43-893f733dae8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 15:21:40.488917 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:40.488845    5034 scope.go:122] "RemoveContainer" containerID="5a98b23f21dbd4904a6335bf41576d892475deaa84bba44752eb4e4ce7324155"
Nov 02 15:21:48.412914 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:21:48.412873    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:21:58.935271 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.935235    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"]
Nov 02 15:21:58.935706 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.935693    5034 cpu_manager.go:414] "containerMap: removing container" podUID="cedbad61-38ea-4234-b56c-f18a7edab309" containerName="revoke-keys"
Nov 02 15:21:58.935756 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.935706    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cedbad61-38ea-4234-b56c-f18a7edab309" containerName="revoke-keys"
Nov 02 15:21:58.935836 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.935796    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="cedbad61-38ea-4234-b56c-f18a7edab309" containerName="revoke-keys"
Nov 02 15:21:58.939159 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.939141    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:58.969947 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.969919    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:21:58.969947 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.969936    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:21:58.970142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.970026    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:21:58.970142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.970034    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:21:58.973744 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.973721    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"]
Nov 02 15:21:58.997689 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.997660    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6db6d0c-qtqbp\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:58.997834 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.997700    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwg59\" (UniqueName: \"kubernetes.io/projected/ae41091f-7d4d-49cc-80c9-24354478f9b3-kube-api-access-mwg59\") pod \"maas-api-revoke-sub-b6db6d0c-qtqbp\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:58.997918 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:58.997820    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6db6d0c-qtqbp\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:59.098364 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:59.098338    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6db6d0c-qtqbp\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:59.098516 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:59.098368    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwg59\" (UniqueName: \"kubernetes.io/projected/ae41091f-7d4d-49cc-80c9-24354478f9b3-kube-api-access-mwg59\") pod \"maas-api-revoke-sub-b6db6d0c-qtqbp\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:59.098516 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:59.098387    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6db6d0c-qtqbp\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:59.099115 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:59.099091    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6db6d0c-qtqbp\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:59.100665 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:59.100644    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6db6d0c-qtqbp\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:59.123163 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:59.123136    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwg59\" (UniqueName: \"kubernetes.io/projected/ae41091f-7d4d-49cc-80c9-24354478f9b3-kube-api-access-mwg59\") pod \"maas-api-revoke-sub-b6db6d0c-qtqbp\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:59.251846 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:59.251816    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:21:59.550396 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:59.550366    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"]
Nov 02 15:21:59.550520 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:21:59.550437    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae41091f_7d4d_49cc_80c9_24354478f9b3.slice/crio-e11606ff58077e01f1f37c055dd23b8860435b5a231943e1dc5c3b69b65353ee WatchSource:0}: Error finding container e11606ff58077e01f1f37c055dd23b8860435b5a231943e1dc5c3b69b65353ee: Status 404 returned error can't find the container with id e11606ff58077e01f1f37c055dd23b8860435b5a231943e1dc5c3b69b65353ee
Nov 02 15:21:59.583764 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:21:59.583734    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp" event={"ID":"ae41091f-7d4d-49cc-80c9-24354478f9b3","Type":"ContainerStarted","Data":"e11606ff58077e01f1f37c055dd23b8860435b5a231943e1dc5c3b69b65353ee"}
Nov 02 15:22:00.590395 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:00.590357    5034 generic.go:356] "Generic (PLEG): container finished" podID="ae41091f-7d4d-49cc-80c9-24354478f9b3" containerID="96705092e940050ba4d1134f5359b098e06173d78015e372632e856de0ae60ce" exitCode=0
Nov 02 15:22:00.590754 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:00.590444    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp" event={"ID":"ae41091f-7d4d-49cc-80c9-24354478f9b3","Type":"ContainerDied","Data":"96705092e940050ba4d1134f5359b098e06173d78015e372632e856de0ae60ce"}
Nov 02 15:22:01.743633 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.743610    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:22:01.821586 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.821550    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae41091f-7d4d-49cc-80c9-24354478f9b3-kube-api-access-mwg59\" (UniqueName: \"kubernetes.io/projected/ae41091f-7d4d-49cc-80c9-24354478f9b3-kube-api-access-mwg59\") pod \"ae41091f-7d4d-49cc-80c9-24354478f9b3\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") "
Nov 02 15:22:01.821739 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.821599    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-xks-ca\") pod \"ae41091f-7d4d-49cc-80c9-24354478f9b3\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") "
Nov 02 15:22:01.821739 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.821662    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-openshift-ca\") pod \"ae41091f-7d4d-49cc-80c9-24354478f9b3\" (UID: \"ae41091f-7d4d-49cc-80c9-24354478f9b3\") "
Nov 02 15:22:01.822102 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.822070    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-openshift-ca" pod "ae41091f-7d4d-49cc-80c9-24354478f9b3" (UID: "ae41091f-7d4d-49cc-80c9-24354478f9b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:01.822241 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.822222    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:01.823641 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.823611    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-xks-ca" pod "ae41091f-7d4d-49cc-80c9-24354478f9b3" (UID: "ae41091f-7d4d-49cc-80c9-24354478f9b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:01.823736 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.823719    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae41091f-7d4d-49cc-80c9-24354478f9b3-kube-api-access-mwg59" pod "ae41091f-7d4d-49cc-80c9-24354478f9b3" (UID: "ae41091f-7d4d-49cc-80c9-24354478f9b3"). InnerVolumeSpecName "kube-api-access-mwg59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:01.923380 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.923326    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwg59\" (UniqueName: \"kubernetes.io/projected/ae41091f-7d4d-49cc-80c9-24354478f9b3-kube-api-access-mwg59\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:01.923380 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:01.923348    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae41091f-7d4d-49cc-80c9-24354478f9b3-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:02.601496 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:02.601459    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp" event={"ID":"ae41091f-7d4d-49cc-80c9-24354478f9b3","Type":"ContainerDied","Data":"e11606ff58077e01f1f37c055dd23b8860435b5a231943e1dc5c3b69b65353ee"}
Nov 02 15:22:02.601496 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:02.601494    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6db6d0c-qtqbp"
Nov 02 15:22:02.601722 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:02.601517    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e11606ff58077e01f1f37c055dd23b8860435b5a231943e1dc5c3b69b65353ee"
Nov 02 15:22:09.320834 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.320798    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"]
Nov 02 15:22:09.321273 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.321258    5034 cpu_manager.go:414] "containerMap: removing container" podUID="ae41091f-7d4d-49cc-80c9-24354478f9b3" containerName="revoke-keys"
Nov 02 15:22:09.321328 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.321273    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae41091f-7d4d-49cc-80c9-24354478f9b3" containerName="revoke-keys"
Nov 02 15:22:09.321388 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.321379    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae41091f-7d4d-49cc-80c9-24354478f9b3" containerName="revoke-keys"
Nov 02 15:22:09.325293 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.325272    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.341032 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.341006    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:22:09.354155 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.354125    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:22:09.354317 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.354253    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:22:09.354317 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.354310    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:22:09.357307 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.357281    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"]
Nov 02 15:22:09.487354 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.487318    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppxjb\" (UniqueName: \"kubernetes.io/projected/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-kube-api-access-ppxjb\") pod \"maas-api-revoke-sub-cbdf5f04-dxgkr\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.487354 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.487358    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbdf5f04-dxgkr\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.487579 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.487374    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbdf5f04-dxgkr\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.588514 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.588430    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppxjb\" (UniqueName: \"kubernetes.io/projected/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-kube-api-access-ppxjb\") pod \"maas-api-revoke-sub-cbdf5f04-dxgkr\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.588514 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.588469    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbdf5f04-dxgkr\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.588514 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.588496    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbdf5f04-dxgkr\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.589207 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.589184    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbdf5f04-dxgkr\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.590764 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.590739    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbdf5f04-dxgkr\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.612978 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.612955    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppxjb\" (UniqueName: \"kubernetes.io/projected/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-kube-api-access-ppxjb\") pod \"maas-api-revoke-sub-cbdf5f04-dxgkr\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.643450 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.643429    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:09.943171 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:22:09.943130    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cc08d3_7585_4c4d_bada_61bfe527b4a5.slice/crio-30cfcc16cdbc1af0fc5117bbcf6844de8a9cbd5394b3badd25f51af9013a5258 WatchSource:0}: Error finding container 30cfcc16cdbc1af0fc5117bbcf6844de8a9cbd5394b3badd25f51af9013a5258: Status 404 returned error can't find the container with id 30cfcc16cdbc1af0fc5117bbcf6844de8a9cbd5394b3badd25f51af9013a5258
Nov 02 15:22:09.943557 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:09.943530    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"]
Nov 02 15:22:10.645712 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:10.645620    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr" event={"ID":"a0cc08d3-7585-4c4d-bada-61bfe527b4a5","Type":"ContainerStarted","Data":"943ce1bf8a62bcbf5cefa642c1a2580941faae3339648f0cd66e10e737b52241"}
Nov 02 15:22:10.645712 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:10.645678    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr" event={"ID":"a0cc08d3-7585-4c4d-bada-61bfe527b4a5","Type":"ContainerStarted","Data":"30cfcc16cdbc1af0fc5117bbcf6844de8a9cbd5394b3badd25f51af9013a5258"}
Nov 02 15:22:10.694743 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:10.694696    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr" podStartSLOduration=1.694683794 podStartE2EDuration="1.694683794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:10.694145251 +0000 UTC m=+5921.445449687" watchObservedRunningTime="2026-11-02 15:22:10.694683794 +0000 UTC m=+5921.445988278"
Nov 02 15:22:11.653450 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:11.653406    5034 generic.go:356] "Generic (PLEG): container finished" podID="a0cc08d3-7585-4c4d-bada-61bfe527b4a5" containerID="943ce1bf8a62bcbf5cefa642c1a2580941faae3339648f0cd66e10e737b52241" exitCode=0
Nov 02 15:22:11.653968 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:11.653501    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr" event={"ID":"a0cc08d3-7585-4c4d-bada-61bfe527b4a5","Type":"ContainerDied","Data":"943ce1bf8a62bcbf5cefa642c1a2580941faae3339648f0cd66e10e737b52241"}
Nov 02 15:22:12.801223 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:12.801191    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:12.920429 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:12.920354    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-openshift-ca\") pod \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") "
Nov 02 15:22:12.920429 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:12.920389    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-xks-ca\") pod \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") "
Nov 02 15:22:12.920675 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:12.920521    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-kube-api-access-ppxjb\" (UniqueName: \"kubernetes.io/projected/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-kube-api-access-ppxjb\") pod \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\" (UID: \"a0cc08d3-7585-4c4d-bada-61bfe527b4a5\") "
Nov 02 15:22:12.920790 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:12.920754    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-openshift-ca" pod "a0cc08d3-7585-4c4d-bada-61bfe527b4a5" (UID: "a0cc08d3-7585-4c4d-bada-61bfe527b4a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:12.920957 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:12.920939    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:12.922543 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:12.922521    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-kube-api-access-ppxjb" pod "a0cc08d3-7585-4c4d-bada-61bfe527b4a5" (UID: "a0cc08d3-7585-4c4d-bada-61bfe527b4a5"). InnerVolumeSpecName "kube-api-access-ppxjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:12.922667 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:12.922626    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-xks-ca" pod "a0cc08d3-7585-4c4d-bada-61bfe527b4a5" (UID: "a0cc08d3-7585-4c4d-bada-61bfe527b4a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:13.022496 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:13.022464    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppxjb\" (UniqueName: \"kubernetes.io/projected/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-kube-api-access-ppxjb\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:13.022496 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:13.022492    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0cc08d3-7585-4c4d-bada-61bfe527b4a5-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:13.669676 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:13.669590    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr"
Nov 02 15:22:13.669886 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:13.669584    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbdf5f04-dxgkr" event={"ID":"a0cc08d3-7585-4c4d-bada-61bfe527b4a5","Type":"ContainerDied","Data":"30cfcc16cdbc1af0fc5117bbcf6844de8a9cbd5394b3badd25f51af9013a5258"}
Nov 02 15:22:13.669886 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:13.669733    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cfcc16cdbc1af0fc5117bbcf6844de8a9cbd5394b3badd25f51af9013a5258"
Nov 02 15:22:18.412112 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:22:18.412065    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:22:19.623089 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.623053    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"]
Nov 02 15:22:19.623698 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.623675    5034 cpu_manager.go:414] "containerMap: removing container" podUID="a0cc08d3-7585-4c4d-bada-61bfe527b4a5" containerName="revoke-keys"
Nov 02 15:22:19.623759 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.623702    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0cc08d3-7585-4c4d-bada-61bfe527b4a5" containerName="revoke-keys"
Nov 02 15:22:19.623901 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.623888    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0cc08d3-7585-4c4d-bada-61bfe527b4a5" containerName="revoke-keys"
Nov 02 15:22:19.627337 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.627314    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.646324 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.646294    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"]
Nov 02 15:22:19.655515 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.655486    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:22:19.656680 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.655919    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:22:19.656860 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.655993    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:22:19.656860 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.656082    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:22:19.767661 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.767628    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d454ce05-kjqk8\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.767847 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.767672    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7t8b\" (UniqueName: \"kubernetes.io/projected/e818a991-1077-4da0-b864-e225a4ab902c-kube-api-access-t7t8b\") pod \"maas-api-revoke-sub-d454ce05-kjqk8\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.767847 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.767751    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d454ce05-kjqk8\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.869018 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.868992    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d454ce05-kjqk8\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.869166 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.869029    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7t8b\" (UniqueName: \"kubernetes.io/projected/e818a991-1077-4da0-b864-e225a4ab902c-kube-api-access-t7t8b\") pod \"maas-api-revoke-sub-d454ce05-kjqk8\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.869166 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.869049    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d454ce05-kjqk8\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.869741 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.869716    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d454ce05-kjqk8\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.871142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.871118    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d454ce05-kjqk8\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.893491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.893439    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7t8b\" (UniqueName: \"kubernetes.io/projected/e818a991-1077-4da0-b864-e225a4ab902c-kube-api-access-t7t8b\") pod \"maas-api-revoke-sub-d454ce05-kjqk8\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:19.940214 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:19.940194    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:20.235864 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:22:20.235835    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode818a991_1077_4da0_b864_e225a4ab902c.slice/crio-a7d593468db08fcb12e581c43e958b3f09622febbdea75921ed0e02d2806776c WatchSource:0}: Error finding container a7d593468db08fcb12e581c43e958b3f09622febbdea75921ed0e02d2806776c: Status 404 returned error can't find the container with id a7d593468db08fcb12e581c43e958b3f09622febbdea75921ed0e02d2806776c
Nov 02 15:22:20.236107 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:20.236085    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"]
Nov 02 15:22:20.743379 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:20.743349    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8" event={"ID":"e818a991-1077-4da0-b864-e225a4ab902c","Type":"ContainerStarted","Data":"de4461e755d59c37ba10a4df5bfa193b306f00242bf667ada94ad3301aa2103d"}
Nov 02 15:22:20.743748 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:20.743407    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8" event={"ID":"e818a991-1077-4da0-b864-e225a4ab902c","Type":"ContainerStarted","Data":"a7d593468db08fcb12e581c43e958b3f09622febbdea75921ed0e02d2806776c"}
Nov 02 15:22:20.786597 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:20.786553    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8" podStartSLOduration=1.786539884 podStartE2EDuration="1.786539884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:20.785338342 +0000 UTC m=+5931.536642779" watchObservedRunningTime="2026-11-02 15:22:20.786539884 +0000 UTC m=+5931.537844320"
Nov 02 15:22:21.750478 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:21.750442    5034 generic.go:356] "Generic (PLEG): container finished" podID="e818a991-1077-4da0-b864-e225a4ab902c" containerID="de4461e755d59c37ba10a4df5bfa193b306f00242bf667ada94ad3301aa2103d" exitCode=0
Nov 02 15:22:21.750888 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:21.750525    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8" event={"ID":"e818a991-1077-4da0-b864-e225a4ab902c","Type":"ContainerDied","Data":"de4461e755d59c37ba10a4df5bfa193b306f00242bf667ada94ad3301aa2103d"}
Nov 02 15:22:22.907151 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:22.907126    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:22.995481 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:22.995454    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-openshift-ca\") pod \"e818a991-1077-4da0-b864-e225a4ab902c\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") "
Nov 02 15:22:22.995634 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:22.995537    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-xks-ca\") pod \"e818a991-1077-4da0-b864-e225a4ab902c\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") "
Nov 02 15:22:22.995634 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:22.995613    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e818a991-1077-4da0-b864-e225a4ab902c-kube-api-access-t7t8b\" (UniqueName: \"kubernetes.io/projected/e818a991-1077-4da0-b864-e225a4ab902c-kube-api-access-t7t8b\") pod \"e818a991-1077-4da0-b864-e225a4ab902c\" (UID: \"e818a991-1077-4da0-b864-e225a4ab902c\") "
Nov 02 15:22:22.995827 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:22.995802    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-openshift-ca" pod "e818a991-1077-4da0-b864-e225a4ab902c" (UID: "e818a991-1077-4da0-b864-e225a4ab902c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:22.995997 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:22.995967    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:22.997517 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:22.997495    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-xks-ca" pod "e818a991-1077-4da0-b864-e225a4ab902c" (UID: "e818a991-1077-4da0-b864-e225a4ab902c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:22.997740 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:22.997722    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e818a991-1077-4da0-b864-e225a4ab902c-kube-api-access-t7t8b" pod "e818a991-1077-4da0-b864-e225a4ab902c" (UID: "e818a991-1077-4da0-b864-e225a4ab902c"). InnerVolumeSpecName "kube-api-access-t7t8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:23.097491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:23.097421    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7t8b\" (UniqueName: \"kubernetes.io/projected/e818a991-1077-4da0-b864-e225a4ab902c-kube-api-access-t7t8b\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:23.097491 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:23.097468    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e818a991-1077-4da0-b864-e225a4ab902c-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:23.770928 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:23.770898    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8"
Nov 02 15:22:23.771092 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:23.770929    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d454ce05-kjqk8" event={"ID":"e818a991-1077-4da0-b864-e225a4ab902c","Type":"ContainerDied","Data":"a7d593468db08fcb12e581c43e958b3f09622febbdea75921ed0e02d2806776c"}
Nov 02 15:22:23.771092 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:23.770974    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7d593468db08fcb12e581c43e958b3f09622febbdea75921ed0e02d2806776c"
Nov 02 15:22:29.952453 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.952418    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"]
Nov 02 15:22:29.952985 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.952900    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e818a991-1077-4da0-b864-e225a4ab902c" containerName="revoke-keys"
Nov 02 15:22:29.952985 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.952914    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e818a991-1077-4da0-b864-e225a4ab902c" containerName="revoke-keys"
Nov 02 15:22:29.952985 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.952987    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e818a991-1077-4da0-b864-e225a4ab902c" containerName="revoke-keys"
Nov 02 15:22:29.956480 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.956455    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:29.973270 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.973247    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:22:29.973740 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.973717    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:22:29.986538 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.986508    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:22:29.986767 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.986569    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:22:29.989506 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:29.989479    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"]
Nov 02 15:22:30.059076 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.059044    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2bbdc2-72kjq\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.059252 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.059083    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2bbdc2-72kjq\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.059252 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.059172    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczb9\" (UniqueName: \"kubernetes.io/projected/f0d1e631-e750-4106-aacf-e15b982c2f76-kube-api-access-sczb9\") pod \"maas-api-revoke-sub-0f2bbdc2-72kjq\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.160238 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.160208    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2bbdc2-72kjq\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.160238 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.160241    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2bbdc2-72kjq\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.160455 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.160267    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczb9\" (UniqueName: \"kubernetes.io/projected/f0d1e631-e750-4106-aacf-e15b982c2f76-kube-api-access-sczb9\") pod \"maas-api-revoke-sub-0f2bbdc2-72kjq\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.160873 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.160849    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f2bbdc2-72kjq\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.162493 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.162470    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f2bbdc2-72kjq\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.184951 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.184918    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczb9\" (UniqueName: \"kubernetes.io/projected/f0d1e631-e750-4106-aacf-e15b982c2f76-kube-api-access-sczb9\") pod \"maas-api-revoke-sub-0f2bbdc2-72kjq\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.269992 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.269971    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:30.575722 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:22:30.575689    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d1e631_e750_4106_aacf_e15b982c2f76.slice/crio-606514d35faa9f31cd398bee89541e29717bcb99ccb4d7a33c55c0788027d07c WatchSource:0}: Error finding container 606514d35faa9f31cd398bee89541e29717bcb99ccb4d7a33c55c0788027d07c: Status 404 returned error can't find the container with id 606514d35faa9f31cd398bee89541e29717bcb99ccb4d7a33c55c0788027d07c
Nov 02 15:22:30.576561 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.576517    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"]
Nov 02 15:22:30.809362 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:30.809330    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq" event={"ID":"f0d1e631-e750-4106-aacf-e15b982c2f76","Type":"ContainerStarted","Data":"606514d35faa9f31cd398bee89541e29717bcb99ccb4d7a33c55c0788027d07c"}
Nov 02 15:22:31.820565 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:31.820525    5034 generic.go:356] "Generic (PLEG): container finished" podID="f0d1e631-e750-4106-aacf-e15b982c2f76" containerID="172498e501b170f34b3d58d585b14e03cd07b860e804b42f07e5e236a387875f" exitCode=0
Nov 02 15:22:31.821109 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:31.820644    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq" event={"ID":"f0d1e631-e750-4106-aacf-e15b982c2f76","Type":"ContainerDied","Data":"172498e501b170f34b3d58d585b14e03cd07b860e804b42f07e5e236a387875f"}
Nov 02 15:22:33.742938 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:33.742908    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:33.891858 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:33.868334    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-xks-ca\") pod \"f0d1e631-e750-4106-aacf-e15b982c2f76\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") "
Nov 02 15:22:33.892217 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:33.892188    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d1e631-e750-4106-aacf-e15b982c2f76-kube-api-access-sczb9\" (UniqueName: \"kubernetes.io/projected/f0d1e631-e750-4106-aacf-e15b982c2f76-kube-api-access-sczb9\") pod \"f0d1e631-e750-4106-aacf-e15b982c2f76\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") "
Nov 02 15:22:33.919894 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:33.919864    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-openshift-ca\") pod \"f0d1e631-e750-4106-aacf-e15b982c2f76\" (UID: \"f0d1e631-e750-4106-aacf-e15b982c2f76\") "
Nov 02 15:22:33.949228 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:33.921166    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-openshift-ca" pod "f0d1e631-e750-4106-aacf-e15b982c2f76" (UID: "f0d1e631-e750-4106-aacf-e15b982c2f76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:34.050061 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:34.050012    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:34.133801 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:34.133424    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d1e631-e750-4106-aacf-e15b982c2f76-kube-api-access-sczb9" pod "f0d1e631-e750-4106-aacf-e15b982c2f76" (UID: "f0d1e631-e750-4106-aacf-e15b982c2f76"). InnerVolumeSpecName "kube-api-access-sczb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:34.135828 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:34.134177    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-xks-ca" pod "f0d1e631-e750-4106-aacf-e15b982c2f76" (UID: "f0d1e631-e750-4106-aacf-e15b982c2f76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:34.135828 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:34.134610    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq" event={"ID":"f0d1e631-e750-4106-aacf-e15b982c2f76","Type":"ContainerDied","Data":"606514d35faa9f31cd398bee89541e29717bcb99ccb4d7a33c55c0788027d07c"}
Nov 02 15:22:34.135828 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:34.134671    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="606514d35faa9f31cd398bee89541e29717bcb99ccb4d7a33c55c0788027d07c"
Nov 02 15:22:34.135828 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:34.134741    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f2bbdc2-72kjq"
Nov 02 15:22:34.151064 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:34.150957    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0d1e631-e750-4106-aacf-e15b982c2f76-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:34.151064 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:34.150981    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sczb9\" (UniqueName: \"kubernetes.io/projected/f0d1e631-e750-4106-aacf-e15b982c2f76-kube-api-access-sczb9\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:34.165170 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:22:34.164110    5034 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d1e631_e750_4106_aacf_e15b982c2f76.slice/memory.min": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d1e631_e750_4106_aacf_e15b982c2f76.slice/memory.min: no such device
Nov 02 15:22:34.390724 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:22:34.390687    5034 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d1e631_e750_4106_aacf_e15b982c2f76.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 15:22:40.313475 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.313440    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"]
Nov 02 15:22:40.313904 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.313890    5034 cpu_manager.go:414] "containerMap: removing container" podUID="f0d1e631-e750-4106-aacf-e15b982c2f76" containerName="revoke-keys"
Nov 02 15:22:40.313953 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.313905    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d1e631-e750-4106-aacf-e15b982c2f76" containerName="revoke-keys"
Nov 02 15:22:40.313994 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.313989    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d1e631-e750-4106-aacf-e15b982c2f76" containerName="revoke-keys"
Nov 02 15:22:40.317344 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.317326    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.333860 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.333835    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:22:40.347203 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.347180    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:22:40.347339 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.347239    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:22:40.347443 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.347409    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:22:40.350062 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.350037    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"]
Nov 02 15:22:40.405030 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.405002    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cf088fb-bwx5g\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.405160 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.405044    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cf088fb-bwx5g\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.405219 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.405173    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b94dh\" (UniqueName: \"kubernetes.io/projected/4af505ca-c834-4224-81e1-568fd19ebf34-kube-api-access-b94dh\") pod \"maas-api-revoke-sub-9cf088fb-bwx5g\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.506220 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.506187    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b94dh\" (UniqueName: \"kubernetes.io/projected/4af505ca-c834-4224-81e1-568fd19ebf34-kube-api-access-b94dh\") pod \"maas-api-revoke-sub-9cf088fb-bwx5g\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.506353 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.506270    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cf088fb-bwx5g\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.506353 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.506297    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cf088fb-bwx5g\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.506946 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.506922    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9cf088fb-bwx5g\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.508566 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.508542    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9cf088fb-bwx5g\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.514324 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.514297    5034 scope.go:122] "RemoveContainer" containerID="5cb140948d393ad6b6552101d61f638f8146776bf9f79b84dfca8b2c9ca83fe8"
Nov 02 15:22:40.530868 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.530846    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b94dh\" (UniqueName: \"kubernetes.io/projected/4af505ca-c834-4224-81e1-568fd19ebf34-kube-api-access-b94dh\") pod \"maas-api-revoke-sub-9cf088fb-bwx5g\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.630304 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.630246    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:40.957112 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:22:40.957077    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af505ca_c834_4224_81e1_568fd19ebf34.slice/crio-5ba85470f8a691d8d244cc6a7c07a19b17a32f91ae783a01ca315d7cf36eeda1 WatchSource:0}: Error finding container 5ba85470f8a691d8d244cc6a7c07a19b17a32f91ae783a01ca315d7cf36eeda1: Status 404 returned error can't find the container with id 5ba85470f8a691d8d244cc6a7c07a19b17a32f91ae783a01ca315d7cf36eeda1
Nov 02 15:22:40.959303 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:40.959278    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"]
Nov 02 15:22:41.176535 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:41.176494    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g" event={"ID":"4af505ca-c834-4224-81e1-568fd19ebf34","Type":"ContainerStarted","Data":"5ba85470f8a691d8d244cc6a7c07a19b17a32f91ae783a01ca315d7cf36eeda1"}
Nov 02 15:22:42.187459 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:42.187355    5034 generic.go:356] "Generic (PLEG): container finished" podID="4af505ca-c834-4224-81e1-568fd19ebf34" containerID="6b1e19b91c7af5118f6873f07a437f5943f635af77a847a494543ac25aa49e06" exitCode=0
Nov 02 15:22:42.187977 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:42.187457    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g" event={"ID":"4af505ca-c834-4224-81e1-568fd19ebf34","Type":"ContainerDied","Data":"6b1e19b91c7af5118f6873f07a437f5943f635af77a847a494543ac25aa49e06"}
Nov 02 15:22:44.288761 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.288678    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g" event={"ID":"4af505ca-c834-4224-81e1-568fd19ebf34","Type":"ContainerDied","Data":"5ba85470f8a691d8d244cc6a7c07a19b17a32f91ae783a01ca315d7cf36eeda1"}
Nov 02 15:22:44.289758 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.289735    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ba85470f8a691d8d244cc6a7c07a19b17a32f91ae783a01ca315d7cf36eeda1"
Nov 02 15:22:44.289945 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.289447    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:44.439425 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.439387    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4af505ca-c834-4224-81e1-568fd19ebf34-kube-api-access-b94dh\" (UniqueName: \"kubernetes.io/projected/4af505ca-c834-4224-81e1-568fd19ebf34-kube-api-access-b94dh\") pod \"4af505ca-c834-4224-81e1-568fd19ebf34\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") "
Nov 02 15:22:44.452806 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.449856    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-openshift-ca\") pod \"4af505ca-c834-4224-81e1-568fd19ebf34\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") "
Nov 02 15:22:44.456193 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.456169    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-xks-ca\") pod \"4af505ca-c834-4224-81e1-568fd19ebf34\" (UID: \"4af505ca-c834-4224-81e1-568fd19ebf34\") "
Nov 02 15:22:44.456650 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.454356    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-openshift-ca" pod "4af505ca-c834-4224-81e1-568fd19ebf34" (UID: "4af505ca-c834-4224-81e1-568fd19ebf34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:44.461996 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.461974    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-xks-ca" pod "4af505ca-c834-4224-81e1-568fd19ebf34" (UID: "4af505ca-c834-4224-81e1-568fd19ebf34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:44.464723 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.464671    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af505ca-c834-4224-81e1-568fd19ebf34-kube-api-access-b94dh" pod "4af505ca-c834-4224-81e1-568fd19ebf34" (UID: "4af505ca-c834-4224-81e1-568fd19ebf34"). InnerVolumeSpecName "kube-api-access-b94dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:44.557385 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.557252    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b94dh\" (UniqueName: \"kubernetes.io/projected/4af505ca-c834-4224-81e1-568fd19ebf34-kube-api-access-b94dh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:44.557385 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.557286    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:44.557385 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:44.557302    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4af505ca-c834-4224-81e1-568fd19ebf34-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:45.294029 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:45.293945    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9cf088fb-bwx5g"
Nov 02 15:22:48.412428 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:22:48.412389    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:22:50.623553 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.623522    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"]
Nov 02 15:22:50.624096 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.624076    5034 cpu_manager.go:414] "containerMap: removing container" podUID="4af505ca-c834-4224-81e1-568fd19ebf34" containerName="revoke-keys"
Nov 02 15:22:50.624155 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.624098    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4af505ca-c834-4224-81e1-568fd19ebf34" containerName="revoke-keys"
Nov 02 15:22:50.624244 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.624231    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="4af505ca-c834-4224-81e1-568fd19ebf34" containerName="revoke-keys"
Nov 02 15:22:50.627492 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.627473    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.657555 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.657531    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:22:50.657555 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.657532    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:22:50.657823 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.657535    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:22:50.657823 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.657744    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:22:50.660432 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.660408    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"]
Nov 02 15:22:50.712244 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.712213    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b514fd-fwjv2\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.712396 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.712272    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnfhh\" (UniqueName: \"kubernetes.io/projected/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-kube-api-access-cnfhh\") pod \"maas-api-revoke-sub-94b514fd-fwjv2\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.712450 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.712398    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b514fd-fwjv2\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.813656 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.813628    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b514fd-fwjv2\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.813857 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.813690    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b514fd-fwjv2\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.813857 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.813726    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnfhh\" (UniqueName: \"kubernetes.io/projected/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-kube-api-access-cnfhh\") pod \"maas-api-revoke-sub-94b514fd-fwjv2\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.814280 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.814253    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94b514fd-fwjv2\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.815977 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.815956    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94b514fd-fwjv2\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.838162 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.838143    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnfhh\" (UniqueName: \"kubernetes.io/projected/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-kube-api-access-cnfhh\") pod \"maas-api-revoke-sub-94b514fd-fwjv2\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:50.941421 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:50.941344    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:51.239533 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:22:51.239503    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe00d3b_4d54_4dfe_ae18_a8b05f8d38cd.slice/crio-91bddf0e0845b9cf24ad76e2760bb6b816d088b578755f6a5b3904b8286b22b6 WatchSource:0}: Error finding container 91bddf0e0845b9cf24ad76e2760bb6b816d088b578755f6a5b3904b8286b22b6: Status 404 returned error can't find the container with id 91bddf0e0845b9cf24ad76e2760bb6b816d088b578755f6a5b3904b8286b22b6
Nov 02 15:22:51.240997 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:51.240968    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"]
Nov 02 15:22:51.327146 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:51.327115    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2" event={"ID":"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd","Type":"ContainerStarted","Data":"91bddf0e0845b9cf24ad76e2760bb6b816d088b578755f6a5b3904b8286b22b6"}
Nov 02 15:22:52.338094 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:52.337993    5034 generic.go:356] "Generic (PLEG): container finished" podID="bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd" containerID="5ef2ea245faf48afdcee1c480e278a09f9c858e6ea6295de752674b876416fe0" exitCode=0
Nov 02 15:22:52.338591 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:52.338188    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2" event={"ID":"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd","Type":"ContainerDied","Data":"5ef2ea245faf48afdcee1c480e278a09f9c858e6ea6295de752674b876416fe0"}
Nov 02 15:22:54.274802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.267935    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:22:54.325835 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.325740    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-xks-ca\") pod \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") "
Nov 02 15:22:54.326148 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.326126    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-kube-api-access-cnfhh\" (UniqueName: \"kubernetes.io/projected/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-kube-api-access-cnfhh\") pod \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") "
Nov 02 15:22:54.326318 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.326302    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-openshift-ca\") pod \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\" (UID: \"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd\") "
Nov 02 15:22:54.327086 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.327062    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-openshift-ca" pod "bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd" (UID: "bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:22:54.349819 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.349790    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-kube-api-access-cnfhh" pod "bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd" (UID: "bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd"). InnerVolumeSpecName "kube-api-access-cnfhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:22:54.409053 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.409021    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-xks-ca" pod "bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd" (UID: "bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:22:54.431546 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.431518    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:54.447676 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.447653    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:54.451799 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.447829    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnfhh\" (UniqueName: \"kubernetes.io/projected/bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd-kube-api-access-cnfhh\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:22:54.479353 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.479321    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2" event={"ID":"bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd","Type":"ContainerDied","Data":"91bddf0e0845b9cf24ad76e2760bb6b816d088b578755f6a5b3904b8286b22b6"}
Nov 02 15:22:54.479596 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.479577    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91bddf0e0845b9cf24ad76e2760bb6b816d088b578755f6a5b3904b8286b22b6"
Nov 02 15:22:54.479802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:22:54.479766    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94b514fd-fwjv2"
Nov 02 15:23:00.981967 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:00.981932    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"]
Nov 02 15:23:00.982433 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:00.982372    5034 cpu_manager.go:414] "containerMap: removing container" podUID="bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd" containerName="revoke-keys"
Nov 02 15:23:00.982433 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:00.982383    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd" containerName="revoke-keys"
Nov 02 15:23:00.982517 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:00.982475    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd" containerName="revoke-keys"
Nov 02 15:23:00.985897 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:00.985877    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.002268 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.002239    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:23:01.012360 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.012328    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ec0c61-tl99m\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.012546 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.012473    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ec0c61-tl99m\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.012546 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.012539    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrkt\" (UniqueName: \"kubernetes.io/projected/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-kube-api-access-rwrkt\") pod \"maas-api-revoke-sub-f8ec0c61-tl99m\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.015488 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.015445    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:01.015488 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.015453    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:23:01.015732 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.015713    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:23:01.018551 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.018527    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"]
Nov 02 15:23:01.113794 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.113750    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ec0c61-tl99m\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.113990 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.113863    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrkt\" (UniqueName: \"kubernetes.io/projected/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-kube-api-access-rwrkt\") pod \"maas-api-revoke-sub-f8ec0c61-tl99m\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.113990 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.113926    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ec0c61-tl99m\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.114517 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.114490    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8ec0c61-tl99m\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.116330 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.116304    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8ec0c61-tl99m\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.137951 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.137928    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrkt\" (UniqueName: \"kubernetes.io/projected/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-kube-api-access-rwrkt\") pod \"maas-api-revoke-sub-f8ec0c61-tl99m\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.299458 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.299390    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:01.601481 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:23:01.601452    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode864a754_efc4_4a44_8ecc_a8bfe3b63e54.slice/crio-aa4f767fba5b1a5de9cf3c8e62bb2b7fb95f0d458183989a2afec82849721de7 WatchSource:0}: Error finding container aa4f767fba5b1a5de9cf3c8e62bb2b7fb95f0d458183989a2afec82849721de7: Status 404 returned error can't find the container with id aa4f767fba5b1a5de9cf3c8e62bb2b7fb95f0d458183989a2afec82849721de7
Nov 02 15:23:01.601952 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:01.601931    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"]
Nov 02 15:23:02.526438 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:02.526056    5034 generic.go:356] "Generic (PLEG): container finished" podID="e864a754-efc4-4a44-8ecc-a8bfe3b63e54" containerID="3b3c0ac57199ea241c8432f7068d4af5c3d8bda0512ffc91052a4db05163d7cc" exitCode=0
Nov 02 15:23:02.526438 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:02.526138    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m" event={"ID":"e864a754-efc4-4a44-8ecc-a8bfe3b63e54","Type":"ContainerDied","Data":"3b3c0ac57199ea241c8432f7068d4af5c3d8bda0512ffc91052a4db05163d7cc"}
Nov 02 15:23:02.526438 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:02.526203    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m" event={"ID":"e864a754-efc4-4a44-8ecc-a8bfe3b63e54","Type":"ContainerStarted","Data":"aa4f767fba5b1a5de9cf3c8e62bb2b7fb95f0d458183989a2afec82849721de7"}
Nov 02 15:23:04.580011 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.579938    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:04.582082 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.582049    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m" event={"ID":"e864a754-efc4-4a44-8ecc-a8bfe3b63e54","Type":"ContainerDied","Data":"aa4f767fba5b1a5de9cf3c8e62bb2b7fb95f0d458183989a2afec82849721de7"}
Nov 02 15:23:04.582290 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.582272    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa4f767fba5b1a5de9cf3c8e62bb2b7fb95f0d458183989a2afec82849721de7"
Nov 02 15:23:04.797114 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.797076    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-openshift-ca\") pod \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") "
Nov 02 15:23:04.806273 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.806245    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-kube-api-access-rwrkt\" (UniqueName: \"kubernetes.io/projected/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-kube-api-access-rwrkt\") pod \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") "
Nov 02 15:23:04.806504 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.806485    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-xks-ca\") pod \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\" (UID: \"e864a754-efc4-4a44-8ecc-a8bfe3b63e54\") "
Nov 02 15:23:04.808506 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.808481    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-openshift-ca" pod "e864a754-efc4-4a44-8ecc-a8bfe3b63e54" (UID: "e864a754-efc4-4a44-8ecc-a8bfe3b63e54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:04.808962 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.808940    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:04.902683 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.902640    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-xks-ca" pod "e864a754-efc4-4a44-8ecc-a8bfe3b63e54" (UID: "e864a754-efc4-4a44-8ecc-a8bfe3b63e54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:04.921803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.913440    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:04.922131 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:04.922092    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-kube-api-access-rwrkt" pod "e864a754-efc4-4a44-8ecc-a8bfe3b63e54" (UID: "e864a754-efc4-4a44-8ecc-a8bfe3b63e54"). InnerVolumeSpecName "kube-api-access-rwrkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:05.021127 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:05.021096    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrkt\" (UniqueName: \"kubernetes.io/projected/e864a754-efc4-4a44-8ecc-a8bfe3b63e54-kube-api-access-rwrkt\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:05.588059 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:05.587974    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8ec0c61-tl99m"
Nov 02 15:23:11.308961 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.308881    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"]
Nov 02 15:23:11.309568 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.309545    5034 cpu_manager.go:414] "containerMap: removing container" podUID="e864a754-efc4-4a44-8ecc-a8bfe3b63e54" containerName="revoke-keys"
Nov 02 15:23:11.309668 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.309569    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e864a754-efc4-4a44-8ecc-a8bfe3b63e54" containerName="revoke-keys"
Nov 02 15:23:11.309734 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.309712    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="e864a754-efc4-4a44-8ecc-a8bfe3b63e54" containerName="revoke-keys"
Nov 02 15:23:11.313197 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.313173    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.344870 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.344830    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:23:11.344870 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.344832    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:11.345088 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.344968    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:23:11.345171 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.345154    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:23:11.349301 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.349273    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"]
Nov 02 15:23:11.388539 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.388509    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55mfq\" (UniqueName: \"kubernetes.io/projected/c3341924-b139-4937-b2b6-394335423100-kube-api-access-55mfq\") pod \"maas-api-revoke-sub-e5667dd1-p8cwb\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.388704 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.388592    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3341924-b139-4937-b2b6-394335423100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5667dd1-p8cwb\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.388704 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.388642    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3341924-b139-4937-b2b6-394335423100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5667dd1-p8cwb\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.489675 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.489648    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55mfq\" (UniqueName: \"kubernetes.io/projected/c3341924-b139-4937-b2b6-394335423100-kube-api-access-55mfq\") pod \"maas-api-revoke-sub-e5667dd1-p8cwb\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.489820 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.489724    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3341924-b139-4937-b2b6-394335423100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5667dd1-p8cwb\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.489820 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.489739    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3341924-b139-4937-b2b6-394335423100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5667dd1-p8cwb\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.490454 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.490435    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3341924-b139-4937-b2b6-394335423100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5667dd1-p8cwb\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.492070 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.492051    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3341924-b139-4937-b2b6-394335423100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5667dd1-p8cwb\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.514266 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.514243    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55mfq\" (UniqueName: \"kubernetes.io/projected/c3341924-b139-4937-b2b6-394335423100-kube-api-access-55mfq\") pod \"maas-api-revoke-sub-e5667dd1-p8cwb\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.643187 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.643120    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:11.943598 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:23:11.943570    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3341924_b139_4937_b2b6_394335423100.slice/crio-60e22061e9234ded11bdd7edc3a9e472eba1af8e6cf2365db4b59c90752652a6 WatchSource:0}: Error finding container 60e22061e9234ded11bdd7edc3a9e472eba1af8e6cf2365db4b59c90752652a6: Status 404 returned error can't find the container with id 60e22061e9234ded11bdd7edc3a9e472eba1af8e6cf2365db4b59c90752652a6
Nov 02 15:23:11.944692 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:11.944664    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"]
Nov 02 15:23:12.629313 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:12.629234    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb" event={"ID":"c3341924-b139-4937-b2b6-394335423100","Type":"ContainerStarted","Data":"f78b18b05784a5f8140232c3f94686e818b3664ec35c4ed8260ad5e46fd7eab0"}
Nov 02 15:23:12.629313 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:12.629291    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb" event={"ID":"c3341924-b139-4937-b2b6-394335423100","Type":"ContainerStarted","Data":"60e22061e9234ded11bdd7edc3a9e472eba1af8e6cf2365db4b59c90752652a6"}
Nov 02 15:23:12.673843 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:12.673797    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb" podStartSLOduration=1.673768393 podStartE2EDuration="1.673768393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:12.670644941 +0000 UTC m=+5983.421949377" watchObservedRunningTime="2026-11-02 15:23:12.673768393 +0000 UTC m=+5983.425072829"
Nov 02 15:23:13.640329 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:13.640293    5034 generic.go:356] "Generic (PLEG): container finished" podID="c3341924-b139-4937-b2b6-394335423100" containerID="f78b18b05784a5f8140232c3f94686e818b3664ec35c4ed8260ad5e46fd7eab0" exitCode=0
Nov 02 15:23:13.640983 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:13.640365    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb" event={"ID":"c3341924-b139-4937-b2b6-394335423100","Type":"ContainerDied","Data":"f78b18b05784a5f8140232c3f94686e818b3664ec35c4ed8260ad5e46fd7eab0"}
Nov 02 15:23:15.472526 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.472495    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:15.632851 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.631862    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3341924-b139-4937-b2b6-394335423100-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3341924-b139-4937-b2b6-394335423100-maas-api-xks-ca\") pod \"c3341924-b139-4937-b2b6-394335423100\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") "
Nov 02 15:23:15.632851 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.631973    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3341924-b139-4937-b2b6-394335423100-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3341924-b139-4937-b2b6-394335423100-maas-api-openshift-ca\") pod \"c3341924-b139-4937-b2b6-394335423100\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") "
Nov 02 15:23:15.632851 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.632061    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3341924-b139-4937-b2b6-394335423100-kube-api-access-55mfq\" (UniqueName: \"kubernetes.io/projected/c3341924-b139-4937-b2b6-394335423100-kube-api-access-55mfq\") pod \"c3341924-b139-4937-b2b6-394335423100\" (UID: \"c3341924-b139-4937-b2b6-394335423100\") "
Nov 02 15:23:15.633889 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.633444    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3341924-b139-4937-b2b6-394335423100-maas-api-openshift-ca" pod "c3341924-b139-4937-b2b6-394335423100" (UID: "c3341924-b139-4937-b2b6-394335423100"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:15.637024 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.636994    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3341924-b139-4937-b2b6-394335423100-kube-api-access-55mfq" pod "c3341924-b139-4937-b2b6-394335423100" (UID: "c3341924-b139-4937-b2b6-394335423100"). InnerVolumeSpecName "kube-api-access-55mfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:15.639734 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.639696    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3341924-b139-4937-b2b6-394335423100-maas-api-xks-ca" pod "c3341924-b139-4937-b2b6-394335423100" (UID: "c3341924-b139-4937-b2b6-394335423100"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:15.662591 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.662572    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb"
Nov 02 15:23:15.662724 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.662571    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5667dd1-p8cwb" event={"ID":"c3341924-b139-4937-b2b6-394335423100","Type":"ContainerDied","Data":"60e22061e9234ded11bdd7edc3a9e472eba1af8e6cf2365db4b59c90752652a6"}
Nov 02 15:23:15.662724 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.662693    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e22061e9234ded11bdd7edc3a9e472eba1af8e6cf2365db4b59c90752652a6"
Nov 02 15:23:15.733242 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.733206    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3341924-b139-4937-b2b6-394335423100-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:15.733242 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.733235    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3341924-b139-4937-b2b6-394335423100-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:15.733242 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:15.733246    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55mfq\" (UniqueName: \"kubernetes.io/projected/c3341924-b139-4937-b2b6-394335423100-kube-api-access-55mfq\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:18.412102 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:23:18.412066    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:23:21.643321 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.643289    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"]
Nov 02 15:23:21.643718 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.643705    5034 cpu_manager.go:414] "containerMap: removing container" podUID="c3341924-b139-4937-b2b6-394335423100" containerName="revoke-keys"
Nov 02 15:23:21.643718 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.643717    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3341924-b139-4937-b2b6-394335423100" containerName="revoke-keys"
Nov 02 15:23:21.643853 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.643841    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3341924-b139-4937-b2b6-394335423100" containerName="revoke-keys"
Nov 02 15:23:21.646935 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.646919    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.663971 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.663949    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:23:21.677362 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.677337    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:21.677539 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.677376    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:23:21.682388 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.682362    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"]
Nov 02 15:23:21.691142 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.691109    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-551a3aef-l5vwb\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.691265 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.691160    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-551a3aef-l5vwb\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.691322 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.691297    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm84g\" (UniqueName: \"kubernetes.io/projected/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-kube-api-access-bm84g\") pod \"maas-api-revoke-sub-551a3aef-l5vwb\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.753371 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.753347    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:23:21.792041 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.792011    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm84g\" (UniqueName: \"kubernetes.io/projected/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-kube-api-access-bm84g\") pod \"maas-api-revoke-sub-551a3aef-l5vwb\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.792174 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.792068    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-551a3aef-l5vwb\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.792174 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.792108    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-551a3aef-l5vwb\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.792818 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.792766    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-551a3aef-l5vwb\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.794406 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.794386    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-551a3aef-l5vwb\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.817186 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.817157    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm84g\" (UniqueName: \"kubernetes.io/projected/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-kube-api-access-bm84g\") pod \"maas-api-revoke-sub-551a3aef-l5vwb\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:21.961364 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:21.961297    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:22.279318 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:23:22.267891    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e78a85_401a_4a59_bdbb_65c9aea7f8d4.slice/crio-b99d253c30d769a3cea9576bafe5a3d9857354acae8e5bddf09ca5636bcdf3f5 WatchSource:0}: Error finding container b99d253c30d769a3cea9576bafe5a3d9857354acae8e5bddf09ca5636bcdf3f5: Status 404 returned error can't find the container with id b99d253c30d769a3cea9576bafe5a3d9857354acae8e5bddf09ca5636bcdf3f5
Nov 02 15:23:22.279318 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:22.270657    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"]
Nov 02 15:23:22.702216 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:22.702172    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb" event={"ID":"03e78a85-401a-4a59-bdbb-65c9aea7f8d4","Type":"ContainerStarted","Data":"7444c5a98a00b9ff176a7160ae793c81ea1882bd03273abd2aedbb73f6be6433"}
Nov 02 15:23:22.702578 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:22.702241    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb" event={"ID":"03e78a85-401a-4a59-bdbb-65c9aea7f8d4","Type":"ContainerStarted","Data":"b99d253c30d769a3cea9576bafe5a3d9857354acae8e5bddf09ca5636bcdf3f5"}
Nov 02 15:23:22.750400 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:22.750353    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb" podStartSLOduration=1.750341538 podStartE2EDuration="1.750341538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:22.749171989 +0000 UTC m=+5993.500476428" watchObservedRunningTime="2026-11-02 15:23:22.750341538 +0000 UTC m=+5993.501645975"
Nov 02 15:23:23.713007 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:23.712963    5034 generic.go:356] "Generic (PLEG): container finished" podID="03e78a85-401a-4a59-bdbb-65c9aea7f8d4" containerID="7444c5a98a00b9ff176a7160ae793c81ea1882bd03273abd2aedbb73f6be6433" exitCode=0
Nov 02 15:23:23.713620 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:23.713127    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb" event={"ID":"03e78a85-401a-4a59-bdbb-65c9aea7f8d4","Type":"ContainerDied","Data":"7444c5a98a00b9ff176a7160ae793c81ea1882bd03273abd2aedbb73f6be6433"}
Nov 02 15:23:25.582800 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.578942    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:25.704802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.704127    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-xks-ca\") pod \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") "
Nov 02 15:23:25.704802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.704269    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-kube-api-access-bm84g\" (UniqueName: \"kubernetes.io/projected/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-kube-api-access-bm84g\") pod \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") "
Nov 02 15:23:25.704802 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.704332    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-openshift-ca\") pod \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\" (UID: \"03e78a85-401a-4a59-bdbb-65c9aea7f8d4\") "
Nov 02 15:23:25.745800 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.745279    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-openshift-ca" pod "03e78a85-401a-4a59-bdbb-65c9aea7f8d4" (UID: "03e78a85-401a-4a59-bdbb-65c9aea7f8d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:25.760039 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.759966    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb" event={"ID":"03e78a85-401a-4a59-bdbb-65c9aea7f8d4","Type":"ContainerDied","Data":"b99d253c30d769a3cea9576bafe5a3d9857354acae8e5bddf09ca5636bcdf3f5"}
Nov 02 15:23:25.760256 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.760236    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b99d253c30d769a3cea9576bafe5a3d9857354acae8e5bddf09ca5636bcdf3f5"
Nov 02 15:23:25.760425 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.760411    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-551a3aef-l5vwb"
Nov 02 15:23:25.762285 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.762262    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-xks-ca" pod "03e78a85-401a-4a59-bdbb-65c9aea7f8d4" (UID: "03e78a85-401a-4a59-bdbb-65c9aea7f8d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:25.764200 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.764160    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-kube-api-access-bm84g" pod "03e78a85-401a-4a59-bdbb-65c9aea7f8d4" (UID: "03e78a85-401a-4a59-bdbb-65c9aea7f8d4"). InnerVolumeSpecName "kube-api-access-bm84g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:25.813335 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.808400    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm84g\" (UniqueName: \"kubernetes.io/projected/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-kube-api-access-bm84g\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:25.813335 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.808433    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:25.813335 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:25.808449    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03e78a85-401a-4a59-bdbb-65c9aea7f8d4-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:31.272204 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:31.272173    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:23:31.272663 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:31.272351    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:23:31.982259 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:31.982226    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"]
Nov 02 15:23:31.983084 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:31.983054    5034 cpu_manager.go:414] "containerMap: removing container" podUID="03e78a85-401a-4a59-bdbb-65c9aea7f8d4" containerName="revoke-keys"
Nov 02 15:23:31.983084 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:31.983081    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e78a85-401a-4a59-bdbb-65c9aea7f8d4" containerName="revoke-keys"
Nov 02 15:23:31.983279 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:31.983178    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e78a85-401a-4a59-bdbb-65c9aea7f8d4" containerName="revoke-keys"
Nov 02 15:23:31.986738 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:31.986720    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.002912 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.002890    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:23:32.015994 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.015967    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:32.016098 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.015970    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:23:32.016098 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.016047    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:23:32.019078 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.019056    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"]
Nov 02 15:23:32.061448 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.061421    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53fc29b8-lwznn\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.061601 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.061453    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854c4\" (UniqueName: \"kubernetes.io/projected/7f644603-a31f-47d2-953f-b1d745fa6044-kube-api-access-854c4\") pod \"maas-api-revoke-sub-53fc29b8-lwznn\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.061679 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.061596    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53fc29b8-lwznn\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.162126 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.162095    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53fc29b8-lwznn\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.162348 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.162171    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53fc29b8-lwznn\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.162348 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.162195    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854c4\" (UniqueName: \"kubernetes.io/projected/7f644603-a31f-47d2-953f-b1d745fa6044-kube-api-access-854c4\") pod \"maas-api-revoke-sub-53fc29b8-lwznn\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.162744 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.162724    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53fc29b8-lwznn\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.164755 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.164718    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53fc29b8-lwznn\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.186912 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.186885    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854c4\" (UniqueName: \"kubernetes.io/projected/7f644603-a31f-47d2-953f-b1d745fa6044-kube-api-access-854c4\") pod \"maas-api-revoke-sub-53fc29b8-lwznn\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.299883 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.299816    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:32.600451 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:23:32.600406    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f644603_a31f_47d2_953f_b1d745fa6044.slice/crio-927989f75a62a57b1f5a8f6202c3f0c90625c75f1b7c62108d2b503a2688844b WatchSource:0}: Error finding container 927989f75a62a57b1f5a8f6202c3f0c90625c75f1b7c62108d2b503a2688844b: Status 404 returned error can't find the container with id 927989f75a62a57b1f5a8f6202c3f0c90625c75f1b7c62108d2b503a2688844b
Nov 02 15:23:32.601809 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.601757    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"]
Nov 02 15:23:32.800498 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:32.800465    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn" event={"ID":"7f644603-a31f-47d2-953f-b1d745fa6044","Type":"ContainerStarted","Data":"927989f75a62a57b1f5a8f6202c3f0c90625c75f1b7c62108d2b503a2688844b"}
Nov 02 15:23:33.811756 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:33.811205    5034 generic.go:356] "Generic (PLEG): container finished" podID="7f644603-a31f-47d2-953f-b1d745fa6044" containerID="9261c8fb1e0056940abd07373ace5428de53d955923f795a5e8b8fa523936f69" exitCode=0
Nov 02 15:23:33.811756 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:33.811286    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn" event={"ID":"7f644603-a31f-47d2-953f-b1d745fa6044","Type":"ContainerDied","Data":"9261c8fb1e0056940abd07373ace5428de53d955923f795a5e8b8fa523936f69"}
Nov 02 15:23:35.826800 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.825933    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:35.919803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.918336    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-openshift-ca\") pod \"7f644603-a31f-47d2-953f-b1d745fa6044\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") "
Nov 02 15:23:35.919803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.918430    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-xks-ca\") pod \"7f644603-a31f-47d2-953f-b1d745fa6044\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") "
Nov 02 15:23:35.919803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.918547    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f644603-a31f-47d2-953f-b1d745fa6044-kube-api-access-854c4\" (UniqueName: \"kubernetes.io/projected/7f644603-a31f-47d2-953f-b1d745fa6044-kube-api-access-854c4\") pod \"7f644603-a31f-47d2-953f-b1d745fa6044\" (UID: \"7f644603-a31f-47d2-953f-b1d745fa6044\") "
Nov 02 15:23:35.967841 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.960012    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-openshift-ca" pod "7f644603-a31f-47d2-953f-b1d745fa6044" (UID: "7f644603-a31f-47d2-953f-b1d745fa6044"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:35.978714 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.978435    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn" event={"ID":"7f644603-a31f-47d2-953f-b1d745fa6044","Type":"ContainerDied","Data":"927989f75a62a57b1f5a8f6202c3f0c90625c75f1b7c62108d2b503a2688844b"}
Nov 02 15:23:35.978714 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.978493    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="927989f75a62a57b1f5a8f6202c3f0c90625c75f1b7c62108d2b503a2688844b"
Nov 02 15:23:35.978714 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.978566    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53fc29b8-lwznn"
Nov 02 15:23:35.989834 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.987281    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f644603-a31f-47d2-953f-b1d745fa6044-kube-api-access-854c4" pod "7f644603-a31f-47d2-953f-b1d745fa6044" (UID: "7f644603-a31f-47d2-953f-b1d745fa6044"). InnerVolumeSpecName "kube-api-access-854c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:35.994849 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:35.990183    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-xks-ca" pod "7f644603-a31f-47d2-953f-b1d745fa6044" (UID: "7f644603-a31f-47d2-953f-b1d745fa6044"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:36.025471 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:36.025198    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854c4\" (UniqueName: \"kubernetes.io/projected/7f644603-a31f-47d2-953f-b1d745fa6044-kube-api-access-854c4\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.025471 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:36.025230    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:36.025471 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:36.025246    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f644603-a31f-47d2-953f-b1d745fa6044-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:42.316707 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.316670    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"]
Nov 02 15:23:42.317166 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.317150    5034 cpu_manager.go:414] "containerMap: removing container" podUID="7f644603-a31f-47d2-953f-b1d745fa6044" containerName="revoke-keys"
Nov 02 15:23:42.317214 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.317167    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f644603-a31f-47d2-953f-b1d745fa6044" containerName="revoke-keys"
Nov 02 15:23:42.317265 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.317254    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f644603-a31f-47d2-953f-b1d745fa6044" containerName="revoke-keys"
Nov 02 15:23:42.320840 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.320819    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.336295 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.336266    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:42.340036 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.340008    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"]
Nov 02 15:23:42.349455 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.349432    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:23:42.378858 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.378831    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-910d19fa-xf6qs\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.379000 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.378948    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-910d19fa-xf6qs\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.379000 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.378974    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plzg6\" (UniqueName: \"kubernetes.io/projected/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-kube-api-access-plzg6\") pod \"maas-api-revoke-sub-910d19fa-xf6qs\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.432034 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.432011    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:23:42.432150 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.432059    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:23:42.480064 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.480023    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-910d19fa-xf6qs\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.480236 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.480123    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-910d19fa-xf6qs\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.480236 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.480152    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plzg6\" (UniqueName: \"kubernetes.io/projected/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-kube-api-access-plzg6\") pod \"maas-api-revoke-sub-910d19fa-xf6qs\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.480835 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.480813    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-910d19fa-xf6qs\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.482831 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.482809    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-910d19fa-xf6qs\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.504600 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.504571    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plzg6\" (UniqueName: \"kubernetes.io/projected/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-kube-api-access-plzg6\") pod \"maas-api-revoke-sub-910d19fa-xf6qs\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.633449 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.633375    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:42.931272 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:23:42.931228    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fa77ae_ac50_4148_a8b3_6a81fb1a7f13.slice/crio-c12e009fd31bd0fa402f7c3df21c7f5d21fd33bc62f0da91f24430c54c43141e WatchSource:0}: Error finding container c12e009fd31bd0fa402f7c3df21c7f5d21fd33bc62f0da91f24430c54c43141e: Status 404 returned error can't find the container with id c12e009fd31bd0fa402f7c3df21c7f5d21fd33bc62f0da91f24430c54c43141e
Nov 02 15:23:42.931650 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:42.931628    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"]
Nov 02 15:23:43.017614 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:43.017575    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs" event={"ID":"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13","Type":"ContainerStarted","Data":"c12e009fd31bd0fa402f7c3df21c7f5d21fd33bc62f0da91f24430c54c43141e"}
Nov 02 15:23:44.028357 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:44.028309    5034 generic.go:356] "Generic (PLEG): container finished" podID="94fa77ae-ac50-4148-a8b3-6a81fb1a7f13" containerID="1466635d454f8a716fd436b97661300167cfe08acb3d7de40de55d64830f3801" exitCode=0
Nov 02 15:23:44.028918 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:44.028417    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs" event={"ID":"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13","Type":"ContainerDied","Data":"1466635d454f8a716fd436b97661300167cfe08acb3d7de40de55d64830f3801"}
Nov 02 15:23:46.249535 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.248817    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs" event={"ID":"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13","Type":"ContainerDied","Data":"c12e009fd31bd0fa402f7c3df21c7f5d21fd33bc62f0da91f24430c54c43141e"}
Nov 02 15:23:46.249535 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.248947    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c12e009fd31bd0fa402f7c3df21c7f5d21fd33bc62f0da91f24430c54c43141e"
Nov 02 15:23:46.276240 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.276214    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:46.381126 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.381086    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-openshift-ca\") pod \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") "
Nov 02 15:23:46.381474 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.381449    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-xks-ca\") pod \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") "
Nov 02 15:23:46.384798 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.381691    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-kube-api-access-plzg6\" (UniqueName: \"kubernetes.io/projected/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-kube-api-access-plzg6\") pod \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\" (UID: \"94fa77ae-ac50-4148-a8b3-6a81fb1a7f13\") "
Nov 02 15:23:46.427706 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.427677    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-openshift-ca" pod "94fa77ae-ac50-4148-a8b3-6a81fb1a7f13" (UID: "94fa77ae-ac50-4148-a8b3-6a81fb1a7f13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:46.460070 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.460031    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-kube-api-access-plzg6" pod "94fa77ae-ac50-4148-a8b3-6a81fb1a7f13" (UID: "94fa77ae-ac50-4148-a8b3-6a81fb1a7f13"). InnerVolumeSpecName "kube-api-access-plzg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:46.486132 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.486064    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plzg6\" (UniqueName: \"kubernetes.io/projected/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-kube-api-access-plzg6\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:46.486132 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.486096    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:46.500598 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.491975    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-xks-ca" pod "94fa77ae-ac50-4148-a8b3-6a81fb1a7f13" (UID: "94fa77ae-ac50-4148-a8b3-6a81fb1a7f13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:46.586639 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:46.586603    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fa77ae-ac50-4148-a8b3-6a81fb1a7f13-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:47.254540 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:47.254509    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-910d19fa-xf6qs"
Nov 02 15:23:48.411742 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:23:48.411708    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:23:52.670286 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.670251    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"]
Nov 02 15:23:52.670766 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.670724    5034 cpu_manager.go:414] "containerMap: removing container" podUID="94fa77ae-ac50-4148-a8b3-6a81fb1a7f13" containerName="revoke-keys"
Nov 02 15:23:52.670766 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.670738    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fa77ae-ac50-4148-a8b3-6a81fb1a7f13" containerName="revoke-keys"
Nov 02 15:23:52.670923 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.670851    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fa77ae-ac50-4148-a8b3-6a81fb1a7f13" containerName="revoke-keys"
Nov 02 15:23:52.674206 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.674184    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.696702 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.696679    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"]
Nov 02 15:23:52.706204 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.706182    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 15:23:52.706328 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.706260    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cwtxc\""
Nov 02 15:23:52.738892 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.738867    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb14f2fc-cd5mq\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.739057 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.739035    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpnmf\" (UniqueName: \"kubernetes.io/projected/5010023d-7aed-4668-bfaa-1cec7c97a8c7-kube-api-access-mpnmf\") pod \"maas-api-revoke-sub-eb14f2fc-cd5mq\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.739124 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.739103    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb14f2fc-cd5mq\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.766338 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.766315    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 15:23:52.766458 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.766430    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 15:23:52.839686 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.839653    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpnmf\" (UniqueName: \"kubernetes.io/projected/5010023d-7aed-4668-bfaa-1cec7c97a8c7-kube-api-access-mpnmf\") pod \"maas-api-revoke-sub-eb14f2fc-cd5mq\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.839847 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.839716    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb14f2fc-cd5mq\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.839847 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.839752    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb14f2fc-cd5mq\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.840383 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.840361    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb14f2fc-cd5mq\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.842266 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.842247    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb14f2fc-cd5mq\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.866827 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.866806    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpnmf\" (UniqueName: \"kubernetes.io/projected/5010023d-7aed-4668-bfaa-1cec7c97a8c7-kube-api-access-mpnmf\") pod \"maas-api-revoke-sub-eb14f2fc-cd5mq\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:52.987052 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:52.987029    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:53.293942 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:23:53.293910    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5010023d_7aed_4668_bfaa_1cec7c97a8c7.slice/crio-13801dbdf6cf3b073c5d8221efdcf357ed699c7dac50422fcb9e9a8e0b49bc97 WatchSource:0}: Error finding container 13801dbdf6cf3b073c5d8221efdcf357ed699c7dac50422fcb9e9a8e0b49bc97: Status 404 returned error can't find the container with id 13801dbdf6cf3b073c5d8221efdcf357ed699c7dac50422fcb9e9a8e0b49bc97
Nov 02 15:23:53.294718 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:53.294696    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"]
Nov 02 15:23:54.319840 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:54.319804    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq" event={"ID":"5010023d-7aed-4668-bfaa-1cec7c97a8c7","Type":"ContainerStarted","Data":"17a573b6af5f315d99f69ac349dfc3682ccfa14020e531a31c00f364b44d894d"}
Nov 02 15:23:54.320387 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:54.320351    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq" event={"ID":"5010023d-7aed-4668-bfaa-1cec7c97a8c7","Type":"ContainerStarted","Data":"13801dbdf6cf3b073c5d8221efdcf357ed699c7dac50422fcb9e9a8e0b49bc97"}
Nov 02 15:23:54.371807 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:54.368586    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq" podStartSLOduration=2.368565779 podStartE2EDuration="2.368565779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:54.368129436 +0000 UTC m=+6025.119433873" watchObservedRunningTime="2026-11-02 15:23:54.368565779 +0000 UTC m=+6025.119870217"
Nov 02 15:23:55.343807 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:55.339455    5034 generic.go:356] "Generic (PLEG): container finished" podID="5010023d-7aed-4668-bfaa-1cec7c97a8c7" containerID="17a573b6af5f315d99f69ac349dfc3682ccfa14020e531a31c00f364b44d894d" exitCode=0
Nov 02 15:23:55.343807 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:55.339532    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq" event={"ID":"5010023d-7aed-4668-bfaa-1cec7c97a8c7","Type":"ContainerDied","Data":"17a573b6af5f315d99f69ac349dfc3682ccfa14020e531a31c00f364b44d894d"}
Nov 02 15:23:56.480746 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.480722    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:56.533294 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.533268    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-xks-ca\") pod \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") "
Nov 02 15:23:56.533439 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.533309    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5010023d-7aed-4668-bfaa-1cec7c97a8c7-kube-api-access-mpnmf\" (UniqueName: \"kubernetes.io/projected/5010023d-7aed-4668-bfaa-1cec7c97a8c7-kube-api-access-mpnmf\") pod \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") "
Nov 02 15:23:56.533439 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.533434    5034 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-openshift-ca\") pod \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\" (UID: \"5010023d-7aed-4668-bfaa-1cec7c97a8c7\") "
Nov 02 15:23:56.533837 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.533813    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-openshift-ca" pod "5010023d-7aed-4668-bfaa-1cec7c97a8c7" (UID: "5010023d-7aed-4668-bfaa-1cec7c97a8c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 15:23:56.535475 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.535448    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-xks-ca" pod "5010023d-7aed-4668-bfaa-1cec7c97a8c7" (UID: "5010023d-7aed-4668-bfaa-1cec7c97a8c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 15:23:56.535572 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.535538    5034 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5010023d-7aed-4668-bfaa-1cec7c97a8c7-kube-api-access-mpnmf" pod "5010023d-7aed-4668-bfaa-1cec7c97a8c7" (UID: "5010023d-7aed-4668-bfaa-1cec7c97a8c7"). InnerVolumeSpecName "kube-api-access-mpnmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 15:23:56.634898 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.634821    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-xks-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:56.634898 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.634884    5034 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpnmf\" (UniqueName: \"kubernetes.io/projected/5010023d-7aed-4668-bfaa-1cec7c97a8c7-kube-api-access-mpnmf\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:56.635085 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:56.634899    5034 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5010023d-7aed-4668-bfaa-1cec7c97a8c7-maas-api-openshift-ca\") on node \"ip-10-0-139-26.ec2.internal\" DevicePath \"\""
Nov 02 15:23:57.352560 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:57.352531    5034 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq"
Nov 02 15:23:57.352747 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:57.352578    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb14f2fc-cd5mq" event={"ID":"5010023d-7aed-4668-bfaa-1cec7c97a8c7","Type":"ContainerDied","Data":"13801dbdf6cf3b073c5d8221efdcf357ed699c7dac50422fcb9e9a8e0b49bc97"}
Nov 02 15:23:57.352747 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:23:57.352655    5034 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13801dbdf6cf3b073c5d8221efdcf357ed699c7dac50422fcb9e9a8e0b49bc97"
Nov 02 15:24:15.324123 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:15.324085    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67755d48d8-2qsfp_0a8b5d62-dc9c-4383-b54c-b26d25258d29/authorino/0.log"
Nov 02 15:24:18.417241 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:24:18.417204    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:24:32.030394 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:32.030355    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"]
Nov 02 15:24:32.043392 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:32.043363    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c5a242a-4mxdk"]
Nov 02 15:24:33.393857 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:33.393820    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2944a2e-3222-4ca7-9da1-6e33855f65a0" path="/var/lib/kubelet/pods/a2944a2e-3222-4ca7-9da1-6e33855f65a0/volumes"
Nov 02 15:24:42.031277 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:42.031203    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"]
Nov 02 15:24:42.042724 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:42.042700    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-929198de-86277"]
Nov 02 15:24:43.393392 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:43.393359    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a85145-3ee8-48b4-95ae-601acdf62a19" path="/var/lib/kubelet/pods/39a85145-3ee8-48b4-95ae-601acdf62a19/volumes"
Nov 02 15:24:48.009665 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:48.009635    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gdvr6_7e043e8d-d6b7-4f76-af4d-46b513dcd3f6/server/0.log"
Nov 02 15:24:48.412017 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:24:48.411940    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:24:49.796675 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:49.796631    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-08ca3420-rlvpf_121cedb8-2e03-4dd6-87ad-3ee40ba1de6e/revoke-keys/0.log"
Nov 02 15:24:49.985117 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:49.985086    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0f2bbdc2-72kjq_f0d1e631-e750-4106-aacf-e15b982c2f76/revoke-keys/0.log"
Nov 02 15:24:50.179113 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:50.179034    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-53fc29b8-lwznn_7f644603-a31f-47d2-953f-b1d745fa6044/revoke-keys/0.log"
Nov 02 15:24:50.371740 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:50.371715    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-551a3aef-l5vwb_03e78a85-401a-4a59-bdbb-65c9aea7f8d4/revoke-keys/0.log"
Nov 02 15:24:50.551998 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:50.551972    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8009dbe9-qbzxf_cedbad61-38ea-4234-b56c-f18a7edab309/revoke-keys/0.log"
Nov 02 15:24:50.746556 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:50.746518    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-910d19fa-xf6qs_94fa77ae-ac50-4148-a8b3-6a81fb1a7f13/revoke-keys/0.log"
Nov 02 15:24:50.917692 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:50.917614    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-94b514fd-fwjv2_bbe00d3b-4d54-4dfe-ae18-a8b05f8d38cd/revoke-keys/0.log"
Nov 02 15:24:51.097245 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:51.097218    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9cf088fb-bwx5g_4af505ca-c834-4224-81e1-568fd19ebf34/revoke-keys/0.log"
Nov 02 15:24:51.285592 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:51.285567    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a387dc28-vtpq4_1c336041-4989-46c4-b03f-4f3a28e6b6de/revoke-keys/0.log"
Nov 02 15:24:51.473726 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:51.473698    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b6db6d0c-qtqbp_ae41091f-7d4d-49cc-80c9-24354478f9b3/revoke-keys/0.log"
Nov 02 15:24:51.663706 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:51.663641    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c06e0c99-qp9nd_c8b95aad-3cb3-4d1c-ba81-948eac5db77d/revoke-keys/0.log"
Nov 02 15:24:51.849069 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:51.849045    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cbdf5f04-dxgkr_a0cc08d3-7585-4c4d-bada-61bfe527b4a5/revoke-keys/0.log"
Nov 02 15:24:52.027675 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:52.027650    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d454ce05-kjqk8_e818a991-1077-4da0-b864-e225a4ab902c/revoke-keys/0.log"
Nov 02 15:24:52.218798 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:52.218753    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e0cfa585-bwlmn_90f5db57-6b4e-445e-be97-a468f73b9511/revoke-keys/0.log"
Nov 02 15:24:52.397664 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:52.397590    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e5667dd1-p8cwb_c3341924-b139-4937-b2b6-394335423100/revoke-keys/0.log"
Nov 02 15:24:52.579343 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:52.579316    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eb14f2fc-cd5mq_5010023d-7aed-4668-bfaa-1cec7c97a8c7/revoke-keys/0.log"
Nov 02 15:24:52.754986 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:52.754958    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f8ec0c61-tl99m_e864a754-efc4-4a44-8ecc-a8bfe3b63e54/revoke-keys/0.log"
Nov 02 15:24:54.822943 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:54.822915    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67755d48d8-2qsfp_0a8b5d62-dc9c-4383-b54c-b26d25258d29/authorino/0.log"
Nov 02 15:24:55.046994 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:55.046968    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cscl8_fc17acb2-e694-4585-8c2b-3fed98a4df14/manager/0.log"
Nov 02 15:24:55.669044 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:55.669017    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7vxsk_a960adf1-ea95-4438-a3dd-fff9cbb60dbd/manager/0.log"
Nov 02 15:24:57.087265 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:57.087240    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv_4983a055-8bd9-43ad-ba61-20cdb90ed101/istio-proxy/0.log"
Nov 02 15:24:58.097195 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:58.097167    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-4s5pw_6b68886e-53d7-470d-8d43-893f733dae8f/istio-proxy/0.log"
Nov 02 15:24:58.293129 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:58.293102    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-8glk8_dcf337cc-18f5-4451-88fe-351cf905dca8/istio-proxy/0.log"
Nov 02 15:24:58.692768 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:58.692739    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-kbj7z_43892ef8-c4cc-414e-bc41-d1626001a0fa/istio-proxy/0.log"
Nov 02 15:24:59.307543 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:59.307513    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-thsm4_565989d2-f1a9-4413-a50d-fd54da149761/istio-proxy/0.log"
Nov 02 15:24:59.955080 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:24:59.955051    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg_50a1f6b2-1feb-4aaa-bbb7-f224ce55f893/istio-proxy/0.log"
Nov 02 15:25:00.362170 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:00.362146    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-699c65dd57-f5zxl_4b0ae7b7-881d-4f6f-92df-7e5536bbeff9/payload-processing/0.log"
Nov 02 15:25:00.573342 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:00.573315    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5466575dd5-gsvt5_d861fac3-737a-4cd7-abc6-4af75b4c1d67/router/0.log"
Nov 02 15:25:04.029185 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:04.029152    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"]
Nov 02 15:25:04.041689 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:04.041664    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0cfa585-bwlmn"]
Nov 02 15:25:05.393542 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:05.393510    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f5db57-6b4e-445e-be97-a468f73b9511" path="/var/lib/kubelet/pods/90f5db57-6b4e-445e-be97-a468f73b9511/volumes"
Nov 02 15:25:11.803162 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:11.803132    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-6fdvk_fb70a37a-41e1-40cd-8238-8af2875c55d6/main/0.log"
Nov 02 15:25:12.206617 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:12.206540    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-nw8tp_d345d401-090c-4d81-a5fc-fbf5ca938209/main/0.log"
Nov 02 15:25:12.798689 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:12.798663    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-dq4mp_e71f9623-d37b-4294-a11b-a408ecc42eb0/main/0.log"
Nov 02 15:25:12.985849 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:12.985826    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-rm4lc_b539abab-f634-47a4-966f-7881e24cce86/main/0.log"
Nov 02 15:25:18.412728 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:25:18.412685    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:25:34.054657 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.054622    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"]
Nov 02 15:25:34.062602 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.062575    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06e0c99-qp9nd"]
Nov 02 15:25:34.212445 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.212414    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gnwq9/must-gather-fmktd"]
Nov 02 15:25:34.212863 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.212846    5034 cpu_manager.go:414] "containerMap: removing container" podUID="5010023d-7aed-4668-bfaa-1cec7c97a8c7" containerName="revoke-keys"
Nov 02 15:25:34.212923 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.212864    5034 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5010023d-7aed-4668-bfaa-1cec7c97a8c7" containerName="revoke-keys"
Nov 02 15:25:34.213001 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.212991    5034 memory_manager.go:365] "RemoveStaleState removing state" podUID="5010023d-7aed-4668-bfaa-1cec7c97a8c7" containerName="revoke-keys"
Nov 02 15:25:34.216434 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.216418    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnwq9/must-gather-fmktd"
Nov 02 15:25:34.241707 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.241687    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnwq9\"/\"openshift-service-ca.crt\""
Nov 02 15:25:34.241707 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.241692    5034 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnwq9\"/\"kube-root-ca.crt\""
Nov 02 15:25:34.242112 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.242098    5034 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gnwq9\"/\"default-dockercfg-8v5rb\""
Nov 02 15:25:34.247093 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.247067    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnwq9/must-gather-fmktd"]
Nov 02 15:25:34.330909 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.330848    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7ca9add-fcc3-4c60-90b1-c2cb6d191b87-must-gather-output\") pod \"must-gather-fmktd\" (UID: \"b7ca9add-fcc3-4c60-90b1-c2cb6d191b87\") " pod="openshift-must-gather-gnwq9/must-gather-fmktd"
Nov 02 15:25:34.330909 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.330897    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdtxt\" (UniqueName: \"kubernetes.io/projected/b7ca9add-fcc3-4c60-90b1-c2cb6d191b87-kube-api-access-bdtxt\") pod \"must-gather-fmktd\" (UID: \"b7ca9add-fcc3-4c60-90b1-c2cb6d191b87\") " pod="openshift-must-gather-gnwq9/must-gather-fmktd"
Nov 02 15:25:34.431838 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.431774    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7ca9add-fcc3-4c60-90b1-c2cb6d191b87-must-gather-output\") pod \"must-gather-fmktd\" (UID: \"b7ca9add-fcc3-4c60-90b1-c2cb6d191b87\") " pod="openshift-must-gather-gnwq9/must-gather-fmktd"
Nov 02 15:25:34.431971 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.431877    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdtxt\" (UniqueName: \"kubernetes.io/projected/b7ca9add-fcc3-4c60-90b1-c2cb6d191b87-kube-api-access-bdtxt\") pod \"must-gather-fmktd\" (UID: \"b7ca9add-fcc3-4c60-90b1-c2cb6d191b87\") " pod="openshift-must-gather-gnwq9/must-gather-fmktd"
Nov 02 15:25:34.432056 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.432036    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b7ca9add-fcc3-4c60-90b1-c2cb6d191b87-must-gather-output\") pod \"must-gather-fmktd\" (UID: \"b7ca9add-fcc3-4c60-90b1-c2cb6d191b87\") " pod="openshift-must-gather-gnwq9/must-gather-fmktd"
Nov 02 15:25:34.457623 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.457602    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdtxt\" (UniqueName: \"kubernetes.io/projected/b7ca9add-fcc3-4c60-90b1-c2cb6d191b87-kube-api-access-bdtxt\") pod \"must-gather-fmktd\" (UID: \"b7ca9add-fcc3-4c60-90b1-c2cb6d191b87\") " pod="openshift-must-gather-gnwq9/must-gather-fmktd"
Nov 02 15:25:34.530529 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.530506    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnwq9/must-gather-fmktd"
Nov 02 15:25:34.832013 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:25:34.831948    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7ca9add_fcc3_4c60_90b1_c2cb6d191b87.slice/crio-5988c2636db302301c5b4f534754306264d8cb7755f09a95b54ce7a0d3f45972 WatchSource:0}: Error finding container 5988c2636db302301c5b4f534754306264d8cb7755f09a95b54ce7a0d3f45972: Status 404 returned error can't find the container with id 5988c2636db302301c5b4f534754306264d8cb7755f09a95b54ce7a0d3f45972
Nov 02 15:25:34.832622 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.832593    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnwq9/must-gather-fmktd"]
Nov 02 15:25:34.833831 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.833812    5034 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 15:25:34.964112 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:34.964076    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnwq9/must-gather-fmktd" event={"ID":"b7ca9add-fcc3-4c60-90b1-c2cb6d191b87","Type":"ContainerStarted","Data":"5988c2636db302301c5b4f534754306264d8cb7755f09a95b54ce7a0d3f45972"}
Nov 02 15:25:35.401352 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:35.401319    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b95aad-3cb3-4d1c-ba81-948eac5db77d" path="/var/lib/kubelet/pods/c8b95aad-3cb3-4d1c-ba81-948eac5db77d/volumes"
Nov 02 15:25:35.972868 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:35.972713    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnwq9/must-gather-fmktd" event={"ID":"b7ca9add-fcc3-4c60-90b1-c2cb6d191b87","Type":"ContainerStarted","Data":"5e8ecf1d687aaf3876e41cd2ce82c8a12489f0a3137c22a64824136333c8a66e"}
Nov 02 15:25:35.972868 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:35.972760    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnwq9/must-gather-fmktd" event={"ID":"b7ca9add-fcc3-4c60-90b1-c2cb6d191b87","Type":"ContainerStarted","Data":"0dc7c6d1ec105007259389695c00687f8d99ae4e3eb1e874ada7f925fda94a34"}
Nov 02 15:25:38.958013 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:38.957982    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zp7ql_00be7ba2-6f06-4ed5-a007-d0ac612177ee/global-pull-secret-syncer/0.log"
Nov 02 15:25:39.137542 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:39.137516    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-75dw4_8816a23c-15b8-4775-8853-6dc6cf7526c5/connection-checker/0.log"
Nov 02 15:25:39.253534 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:39.253509    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pm5xb_18d90fc8-23b1-4fca-a9e2-83eb8f4786d4/connection-checker/0.log"
Nov 02 15:25:39.370666 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:39.370641    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zlk72_35f03797-afb7-4aa5-a841-c665e99feaa4/connection-checker/0.log"
Nov 02 15:25:39.798809 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:39.797914    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rfknw_5d26dd5a-118a-4d50-9661-22feeb24d158/konnectivity-agent/0.log"
Nov 02 15:25:40.051847 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:40.049745    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-26.ec2.internal_3694267f060f6a7f596609ec14c03ba4/haproxy/0.log"
Nov 02 15:25:40.562804 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:40.560933    5034 scope.go:122] "RemoveContainer" containerID="c10b43294686bd0b34fb450ee406cee1f0948177019d83522774496e3064cff7"
Nov 02 15:25:40.616078 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:40.613880    5034 scope.go:122] "RemoveContainer" containerID="f08223b96156eff039f41ed0d373ea1dcbc6ba4672859d6d6e1a234b72760466"
Nov 02 15:25:43.167531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:43.167481    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gdvr6_7e043e8d-d6b7-4f76-af4d-46b513dcd3f6/server/0.log"
Nov 02 15:25:45.034803 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:45.034019    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gnwq9/must-gather-fmktd" podStartSLOduration=10.148312589 podStartE2EDuration="11.033998125s" totalImagesPullingTime="885.685536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:36.03225302 +0000 UTC m=+6126.783557467" watchObservedRunningTime="2026-11-02 15:25:45.033998125 +0000 UTC m=+6135.785302564"
Nov 02 15:25:45.035589 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:45.035529    5034 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"]
Nov 02 15:25:45.050369 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:45.050332    5034 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a387dc28-vtpq4"]
Nov 02 15:25:45.401455 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:45.401357    5034 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c336041-4989-46c4-b03f-4f3a28e6b6de" path="/var/lib/kubelet/pods/1c336041-4989-46c4-b03f-4f3a28e6b6de/volumes"
Nov 02 15:25:45.491809 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:45.490552    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67755d48d8-2qsfp_0a8b5d62-dc9c-4383-b54c-b26d25258d29/authorino/0.log"
Nov 02 15:25:45.742224 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:45.742192    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gdvr6_7e043e8d-d6b7-4f76-af4d-46b513dcd3f6/server/0.log"
Nov 02 15:25:46.033192 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:46.033120    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cscl8_fc17acb2-e694-4585-8c2b-3fed98a4df14/manager/0.log"
Nov 02 15:25:46.462445 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:46.462178    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7vxsk_a960adf1-ea95-4438-a3dd-fff9cbb60dbd/manager/0.log"
Nov 02 15:25:48.345809 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.344182    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-gdvr6_7e043e8d-d6b7-4f76-af4d-46b513dcd3f6/server/0.log"
Nov 02 15:25:48.383030 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.382987    5034 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"]
Nov 02 15:25:48.388047 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.388016    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.413516 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.412484    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"]
Nov 02 15:25:48.425952 ip-10-0-139-26 kubenswrapper[5034]: E1102 15:25:48.422911    5034 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-2m5tg" containerName="extractor"
Nov 02 15:25:48.513884 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.513847    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-lib-modules\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.513884 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.513889    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-proc\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.514156 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.513970    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-sys\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.514156 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.514045    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-podres\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.514156 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.514136    5034 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52lg5\" (UniqueName: \"kubernetes.io/projected/01be06bf-ea4a-4bde-bd5a-f9044a70b065-kube-api-access-52lg5\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.615362 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.615287    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52lg5\" (UniqueName: \"kubernetes.io/projected/01be06bf-ea4a-4bde-bd5a-f9044a70b065-kube-api-access-52lg5\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.615362 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.615340    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-lib-modules\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.615593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.615376    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-proc\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.615593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.615410    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-sys\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.615593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.615486    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-proc\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.615593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.615521    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-lib-modules\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.615593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.615541    5034 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-podres\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.615593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.615585    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-sys\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.615593 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.615594    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/01be06bf-ea4a-4bde-bd5a-f9044a70b065-podres\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.640107 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.640065    5034 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52lg5\" (UniqueName: \"kubernetes.io/projected/01be06bf-ea4a-4bde-bd5a-f9044a70b065-kube-api-access-52lg5\") pod \"perf-node-gather-daemonset-nwrgz\" (UID: \"01be06bf-ea4a-4bde-bd5a-f9044a70b065\") " pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.728983 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.728950    5034 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:48.969551 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:48.969447    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-nnxtg_2ca56ffb-997f-4edb-9efa-0609042e8f08/cluster-monitoring-operator/0.log"
Nov 02 15:25:49.037165 ip-10-0-139-26 kubenswrapper[5034]: W1102 15:25:49.037115    5034 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod01be06bf_ea4a_4bde_bd5a_f9044a70b065.slice/crio-ebd4062cc16a3d978b1cd753704a06dcd9bf2eda68f555271dbbd91b8d5ad9d1 WatchSource:0}: Error finding container ebd4062cc16a3d978b1cd753704a06dcd9bf2eda68f555271dbbd91b8d5ad9d1: Status 404 returned error can't find the container with id ebd4062cc16a3d978b1cd753704a06dcd9bf2eda68f555271dbbd91b8d5ad9d1
Nov 02 15:25:49.037528 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:49.037503    5034 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"]
Nov 02 15:25:49.113797 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:49.113050    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz" event={"ID":"01be06bf-ea4a-4bde-bd5a-f9044a70b065","Type":"ContainerStarted","Data":"ebd4062cc16a3d978b1cd753704a06dcd9bf2eda68f555271dbbd91b8d5ad9d1"}
Nov 02 15:25:49.671120 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:49.671003    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-btv25_9afd0929-6d3a-418c-bd2e-902443e20a7a/node-exporter/0.log"
Nov 02 15:25:49.783427 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:49.783397    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-btv25_9afd0929-6d3a-418c-bd2e-902443e20a7a/kube-rbac-proxy/0.log"
Nov 02 15:25:49.896114 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:49.896084    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-btv25_9afd0929-6d3a-418c-bd2e-902443e20a7a/init-textfile/0.log"
Nov 02 15:25:50.112962 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:50.112931    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv_4983a055-8bd9-43ad-ba61-20cdb90ed101/istio-proxy/0.log"
Nov 02 15:25:50.121045 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:50.121003    5034 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz" event={"ID":"01be06bf-ea4a-4bde-bd5a-f9044a70b065","Type":"ContainerStarted","Data":"5c0e594673e410e435dbe53f0f2839b889f49473df2f8036d9f854927bbda35f"}
Nov 02 15:25:50.121222 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:50.121082    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:50.164377 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:50.164323    5034 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz" podStartSLOduration=2.1643073250000002 podStartE2EDuration="2.164307325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 15:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:50.162510658 +0000 UTC m=+6140.913815108" watchObservedRunningTime="2026-11-02 15:25:50.164307325 +0000 UTC m=+6140.915611763"
Nov 02 15:25:51.756540 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:51.756506    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-4s5pw_6b68886e-53d7-470d-8d43-893f733dae8f/istio-proxy/0.log"
Nov 02 15:25:51.822506 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:51.822475    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-8glk8_dcf337cc-18f5-4451-88fe-351cf905dca8/istio-proxy/0.log"
Nov 02 15:25:51.955801 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:51.955379    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-kbj7z_43892ef8-c4cc-414e-bc41-d1626001a0fa/istio-proxy/0.log"
Nov 02 15:25:52.143519 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:52.143456    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-rmw2p_aae1ab7a-b2c1-4716-a1af-82815b203327/prometheus-operator-admission-webhook/0.log"
Nov 02 15:25:52.166654 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:52.166631    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-thsm4_565989d2-f1a9-4413-a50d-fd54da149761/istio-proxy/0.log"
Nov 02 15:25:52.365900 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:52.365870    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg_50a1f6b2-1feb-4aaa-bbb7-f224ce55f893/istio-proxy/0.log"
Nov 02 15:25:52.501375 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:52.501343    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-699c65dd57-f5zxl_4b0ae7b7-881d-4f6f-92df-7e5536bbeff9/payload-processing/0.log"
Nov 02 15:25:52.574932 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:52.574907    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5466575dd5-gsvt5_d861fac3-737a-4cd7-abc6-4af75b4c1d67/router/0.log"
Nov 02 15:25:54.629715 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:54.629691    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-jc4gv_b34b2c6f-210f-4256-8383-16f6a3a360bb/networking-console-plugin/0.log"
Nov 02 15:25:55.679574 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:55.679548    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/2.log"
Nov 02 15:25:55.685793 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:55.685757    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-zfzrk_398a3f5c-7cbe-448d-8f16-3360c18a38bf/console-operator/3.log"
Nov 02 15:25:56.140721 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:56.140570    5034 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gnwq9/perf-node-gather-daemonset-nwrgz"
Nov 02 15:25:56.584570 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:56.584536    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7b7f467c7c-lmsjl_85b0ee54-727d-4550-90b3-3df145901123/console/0.log"
Nov 02 15:25:56.653767 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:56.653738    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-67755d48d8-2qsfp_0a8b5d62-dc9c-4383-b54c-b26d25258d29/authorino/0.log"
Nov 02 15:25:56.692546 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:56.692512    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-cscl8_fc17acb2-e694-4585-8c2b-3fed98a4df14/manager/0.log"
Nov 02 15:25:56.720705 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:56.720677    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-5tvk9_1d0dbccc-3b95-4872-988a-b8df4e6b02f0/download-server/0.log"
Nov 02 15:25:56.920852 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:56.917792    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7vxsk_a960adf1-ea95-4438-a3dd-fff9cbb60dbd/manager/0.log"
Nov 02 15:25:57.845654 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:25:57.845622    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-np42z_4150923a-78f1-4f52-a0d5-271ec942bd1b/volume-data-source-validator/0.log"
Nov 02 15:26:00.370365 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:00.370335    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fzlvp_63881d4e-a3bf-4948-a369-c892baba23a8/dns/0.log"
Nov 02 15:26:00.481085 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:00.481059    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fzlvp_63881d4e-a3bf-4948-a369-c892baba23a8/kube-rbac-proxy/0.log"
Nov 02 15:26:01.071265 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:01.071237    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t6vhd_26f2db88-b241-451f-8f1e-9f8c68ad0e73/dns-node-resolver/0.log"
Nov 02 15:26:01.242463 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:01.242433    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv_4983a055-8bd9-43ad-ba61-20cdb90ed101/istio-proxy/0.log"
Nov 02 15:26:02.219018 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:02.218981    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-57cccf57dc-xn8xj_7a0f79c9-79d9-41da-9617-ef20d42df29d/registry/0.log"
Nov 02 15:26:02.348531 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:02.348504    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-29jl9_c7b5b1d2-b605-4f94-aad7-c7f01d84d6bd/node-ca/0.log"
Nov 02 15:26:02.876498 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:02.876465    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-4s5pw_6b68886e-53d7-470d-8d43-893f733dae8f/istio-proxy/0.log"
Nov 02 15:26:02.942165 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:02.942136    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-8glk8_dcf337cc-18f5-4451-88fe-351cf905dca8/istio-proxy/0.log"
Nov 02 15:26:03.071040 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:03.071002    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-kbj7z_43892ef8-c4cc-414e-bc41-d1626001a0fa/istio-proxy/0.log"
Nov 02 15:26:03.304913 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:03.304877    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-thsm4_565989d2-f1a9-4413-a50d-fd54da149761/istio-proxy/0.log"
Nov 02 15:26:03.509065 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:03.509038    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-55f56bc75f-swzhg_50a1f6b2-1feb-4aaa-bbb7-f224ce55f893/istio-proxy/0.log"
Nov 02 15:26:03.655502 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:03.655434    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-699c65dd57-f5zxl_4b0ae7b7-881d-4f6f-92df-7e5536bbeff9/payload-processing/0.log"
Nov 02 15:26:03.726704 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:03.726679    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5466575dd5-gsvt5_d861fac3-737a-4cd7-abc6-4af75b4c1d67/router/0.log"
Nov 02 15:26:04.287526 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:04.287490    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7845f94dd5qfcdv_4983a055-8bd9-43ad-ba61-20cdb90ed101/istio-proxy/0.log"
Nov 02 15:26:06.754599 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:06.754568    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-4s5pw_6b68886e-53d7-470d-8d43-893f733dae8f/istio-proxy/0.log"
Nov 02 15:26:06.881077 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:06.881032    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-8glk8_dcf337cc-18f5-4451-88fe-351cf905dca8/istio-proxy/0.log"
Nov 02 15:26:07.115974 ip-10-0-139-26 kubenswrapper[5034]: I1102 15:26:07.115895    5034 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-kbj7z_43892ef8-c4cc-414e-bc41-d1626001a0fa/istio-proxy/0.log"